Note: This is test shard 3 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest
[ RUN ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 14:23:10.503630 5549 test_util.cc:268] Using random seed: 1471757980
I20240502 14:23:10.514417 5549 ts_itest-base.cc:112] Starting cluster with:
I20240502 14:23:10.514483 5549 ts_itest-base.cc:113] --------------
I20240502 14:23:10.514526 5549 ts_itest-base.cc:114] 8 tablet servers
I20240502 14:23:10.514588 5549 ts_itest-base.cc:115] 3 replicas per TS
I20240502 14:23:10.514648 5549 ts_itest-base.cc:116] --------------
2024-05-02T14:23:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:23:10Z Disabled control of system clock
I20240502 14:23:10.536875 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.107.126:34027
--webserver_interface=127.5.107.126
--webserver_port=0
--builtin_ntp_servers=127.5.107.84:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.107.126:34027
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 14:23:10.820437 5561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:23:10.820701 5561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:23:10.820820 5561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:23:10.827416 5561 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:23:10.827525 5561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:23:10.827594 5561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:23:10.827668 5561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:23:10.827728 5561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:23:10.835214 5561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.5.107.126:34027
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.107.126:34027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.107.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:23:10.836383 5561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:23:10.838042 5561 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:23:10.847321 5566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:23:10.847919 5567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:23:10.848471 5569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:23:10.848722 5561 server_base.cc:1034] running on GCE node
I20240502 14:23:10.849357 5561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:23:10.850479 5561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:23:10.851655 5561 hybrid_clock.cc:648] HybridClock initialized: now 1714659790851620 us; error 33 us; skew 500 ppm
I20240502 14:23:10.851930 5561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:23:10.858268 5561 webserver.cc:460] Webserver started at http://127.5.107.126:37397/ using document root <none> and password file <none>
I20240502 14:23:10.858877 5561 fs_manager.cc:359] Metadata directory not provided
I20240502 14:23:10.858971 5561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:23:10.859230 5561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:23:10.861479 5561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ef40d42cfd5a4bac8f87b77b2c218537"
format_stamp: "Formatted at 2024-05-02 14:23:10 on dist-test-slave-2cw1"
I20240502 14:23:10.862164 5561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ef40d42cfd5a4bac8f87b77b2c218537"
format_stamp: "Formatted at 2024-05-02 14:23:10 on dist-test-slave-2cw1"
I20240502 14:23:10.866477 5561 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20240502 14:23:10.869163 5575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:10.870240 5561 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:23:10.870380 5561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ef40d42cfd5a4bac8f87b77b2c218537"
format_stamp: "Formatted at 2024-05-02 14:23:10 on dist-test-slave-2cw1"
I20240502 14:23:10.870538 5561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:23:10.890596 5561 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:23:10.891343 5561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:23:10.891592 5561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:23:10.907634 5561 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.126:34027
I20240502 14:23:10.907651 5626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.126:34027 every 8 connection(s)
I20240502 14:23:10.908761 5561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 14:23:10.912041 5627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:10.918777 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 5561
I20240502 14:23:10.919009 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 14:23:10.920841 5627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef40d42cfd5a4bac8f87b77b2c218537: Bootstrap starting.
I20240502 14:23:10.924547 5627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef40d42cfd5a4bac8f87b77b2c218537: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:10.925925 5627 log.cc:824] T 00000000000000000000000000000000 P ef40d42cfd5a4bac8f87b77b2c218537: Log is configured to *not* fsync() on all Append() calls
I20240502 14:23:10.928606 5627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef40d42cfd5a4bac8f87b77b2c218537: No bootstrap required, opened a new log
I20240502 14:23:10.936983 5627 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef40d42cfd5a4bac8f87b77b2c218537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef40d42cfd5a4bac8f87b77b2c218537" member_type: VOTER last_known_addr { host: "127.5.107.126" port: 34027 } }
I20240502 14:23:10.937407 5627 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef40d42cfd5a4bac8f87b77b2c218537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:10.937510 5627 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef40d42cfd5a4bac8f87b77b2c218537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef40d42cfd5a4bac8f87b77b2c218537, State: Initialized, Role: FOLLOWER
I20240502 14:23:10.938123 5627 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef40d42cfd5a4bac8f87b77b2c218537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef40d42cfd5a4bac8f87b77b2c218537" member_type: VOTER last_known_addr { host: "127.5.107.126" port: 34027 } }
I20240502 14:23:10.938344 5627 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef40d42cfd5a4bac8f87b77b2c218537 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:23:10.938457 5627 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef40d42cfd5a4bac8f87b77b2c218537 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:23:10.938619 5627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef40d42cfd5a4bac8f87b77b2c218537 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:10.940793 5627 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef40d42cfd5a4bac8f87b77b2c218537 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef40d42cfd5a4bac8f87b77b2c218537" member_type: VOTER last_known_addr { host: "127.5.107.126" port: 34027 } }
I20240502 14:23:10.941435 5627 leader_election.cc:302] T 00000000000000000000000000000000 P ef40d42cfd5a4bac8f87b77b2c218537 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef40d42cfd5a4bac8f87b77b2c218537; no voters:
I20240502 14:23:10.942212 5627 leader_election.cc:288] T 00000000000000000000000000000000 P ef40d42cfd5a4bac8f87b77b2c218537 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:23:10.942341 5632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef40d42cfd5a4bac8f87b77b2c218537 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:23:10.942940 5632 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef40d42cfd5a4bac8f87b77b2c218537 [term 1 LEADER]: Becoming Leader. State: Replica: ef40d42cfd5a4bac8f87b77b2c218537, State: Running, Role: LEADER
I20240502 14:23:10.943497 5632 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef40d42cfd5a4bac8f87b77b2c218537 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef40d42cfd5a4bac8f87b77b2c218537" member_type: VOTER last_known_addr { host: "127.5.107.126" port: 34027 } }
I20240502 14:23:10.944474 5627 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef40d42cfd5a4bac8f87b77b2c218537 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:23:10.946132 5633 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef40d42cfd5a4bac8f87b77b2c218537 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef40d42cfd5a4bac8f87b77b2c218537" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef40d42cfd5a4bac8f87b77b2c218537" member_type: VOTER last_known_addr { host: "127.5.107.126" port: 34027 } } }
I20240502 14:23:10.946605 5633 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef40d42cfd5a4bac8f87b77b2c218537 [sys.catalog]: This master's current role is: LEADER
I20240502 14:23:10.947239 5634 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef40d42cfd5a4bac8f87b77b2c218537 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef40d42cfd5a4bac8f87b77b2c218537. Latest consensus state: current_term: 1 leader_uuid: "ef40d42cfd5a4bac8f87b77b2c218537" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef40d42cfd5a4bac8f87b77b2c218537" member_type: VOTER last_known_addr { host: "127.5.107.126" port: 34027 } } }
I20240502 14:23:10.947517 5634 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef40d42cfd5a4bac8f87b77b2c218537 [sys.catalog]: This master's current role is: LEADER
I20240502 14:23:10.949627 5637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:23:10.954880 5637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:23:10.962327 5637 catalog_manager.cc:1347] Generated new cluster ID: a6613a4bb53b459091d0bf6e2a424ce2
I20240502 14:23:10.962421 5637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:23:10.985780 5637 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:23:10.986996 5637 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:23:10.996604 5637 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ef40d42cfd5a4bac8f87b77b2c218537: Generated new TSK 0
I20240502 14:23:10.997476 5637 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:23:11.013247 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.65:0
--local_ip_for_outbound_sockets=127.5.107.65
--webserver_interface=127.5.107.65
--webserver_port=0
--tserver_master_addrs=127.5.107.126:34027
--builtin_ntp_servers=127.5.107.84:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 14:23:11.306166 5652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:23:11.306401 5652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:23:11.306535 5652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:23:11.313330 5652 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:23:11.313450 5652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:23:11.313606 5652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.65
I20240502 14:23:11.321946 5652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.107.65
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.5.107.126:34027
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.65
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:23:11.323216 5652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:23:11.324893 5652 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:23:11.335796 5657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:23:11.336382 5660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:23:11.336934 5658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:23:11.337154 5652 server_base.cc:1034] running on GCE node
I20240502 14:23:11.337678 5652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:23:11.338472 5652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:23:11.339663 5652 hybrid_clock.cc:648] HybridClock initialized: now 1714659791339618 us; error 32 us; skew 500 ppm
I20240502 14:23:11.339927 5652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:23:11.341928 5652 webserver.cc:460] Webserver started at http://127.5.107.65:40073/ using document root <none> and password file <none>
I20240502 14:23:11.342568 5652 fs_manager.cc:359] Metadata directory not provided
I20240502 14:23:11.342674 5652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:23:11.342957 5652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:23:11.345317 5652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "1dbf482780eb47e8933983e4f0fd0444"
format_stamp: "Formatted at 2024-05-02 14:23:11 on dist-test-slave-2cw1"
I20240502 14:23:11.346081 5652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "1dbf482780eb47e8933983e4f0fd0444"
format_stamp: "Formatted at 2024-05-02 14:23:11 on dist-test-slave-2cw1"
I20240502 14:23:11.350405 5652 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20240502 14:23:11.353412 5666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:11.354540 5652 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:23:11.354676 5652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "1dbf482780eb47e8933983e4f0fd0444"
format_stamp: "Formatted at 2024-05-02 14:23:11 on dist-test-slave-2cw1"
I20240502 14:23:11.354892 5652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:23:11.374014 5652 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:23:11.374632 5652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:23:11.374852 5652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:23:11.375742 5652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:23:11.376957 5652 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:23:11.377032 5652 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:11.377141 5652 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:23:11.377231 5652 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20240502 14:23:11.405110 5652 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.65:44831
I20240502 14:23:11.405169 5778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.65:44831 every 8 connection(s)
I20240502 14:23:11.406309 5652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 14:23:11.408478 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 5652
I20240502 14:23:11.408682 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 14:23:11.411998 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.66:0
--local_ip_for_outbound_sockets=127.5.107.66
--webserver_interface=127.5.107.66
--webserver_port=0
--tserver_master_addrs=127.5.107.126:34027
--builtin_ntp_servers=127.5.107.84:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 14:23:11.417909 5779 heartbeater.cc:347] Connected to a master server at 127.5.107.126:34027
I20240502 14:23:11.418157 5779 heartbeater.cc:464] Registering TS with master...
I20240502 14:23:11.418852 5779 heartbeater.cc:511] Master 127.5.107.126:34027 requested a full tablet report, sending...
I20240502 14:23:11.420522 5592 ts_manager.cc:194] Registered new tserver with Master: 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831)
I20240502 14:23:11.422039 5592 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.65:41833
W20240502 14:23:11.698922 5782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:23:11.699198 5782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:23:11.699343 5782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:23:11.706212 5782 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:23:11.706326 5782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:23:11.706467 5782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.66
I20240502 14:23:11.714390 5782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.107.66
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.5.107.126:34027
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.66
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:23:11.715543 5782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:23:11.717275 5782 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:23:11.728181 5787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:23:11.728730 5790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:23:11.729350 5788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:23:11.729532 5782 server_base.cc:1034] running on GCE node
I20240502 14:23:11.730019 5782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:23:11.730739 5782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:23:11.731914 5782 hybrid_clock.cc:648] HybridClock initialized: now 1714659791731881 us; error 28 us; skew 500 ppm
I20240502 14:23:11.732198 5782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:23:11.734191 5782 webserver.cc:460] Webserver started at http://127.5.107.66:33203/ using document root <none> and password file <none>
I20240502 14:23:11.734845 5782 fs_manager.cc:359] Metadata directory not provided
I20240502 14:23:11.734972 5782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:23:11.735270 5782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:23:11.737676 5782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "83e62d4e5f8a4316bc1bcb8c18458081"
format_stamp: "Formatted at 2024-05-02 14:23:11 on dist-test-slave-2cw1"
I20240502 14:23:11.738379 5782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "83e62d4e5f8a4316bc1bcb8c18458081"
format_stamp: "Formatted at 2024-05-02 14:23:11 on dist-test-slave-2cw1"
I20240502 14:23:11.742844 5782 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 14:23:11.745811 5796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:11.746953 5782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:23:11.747089 5782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "83e62d4e5f8a4316bc1bcb8c18458081"
format_stamp: "Formatted at 2024-05-02 14:23:11 on dist-test-slave-2cw1"
I20240502 14:23:11.747246 5782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:23:11.760332 5782 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:23:11.761019 5782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:23:11.761317 5782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:23:11.762159 5782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:23:11.763427 5782 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:23:11.763507 5782 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:11.763586 5782 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:23:11.763655 5782 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:11.792542 5782 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.66:34015
I20240502 14:23:11.792598 5908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.66:34015 every 8 connection(s)
I20240502 14:23:11.793761 5782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 14:23:11.796634 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 5782
I20240502 14:23:11.796802 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 14:23:11.800066 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.67:0
--local_ip_for_outbound_sockets=127.5.107.67
--webserver_interface=127.5.107.67
--webserver_port=0
--tserver_master_addrs=127.5.107.126:34027
--builtin_ntp_servers=127.5.107.84:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 14:23:11.806113 5909 heartbeater.cc:347] Connected to a master server at 127.5.107.126:34027
I20240502 14:23:11.806371 5909 heartbeater.cc:464] Registering TS with master...
I20240502 14:23:11.807013 5909 heartbeater.cc:511] Master 127.5.107.126:34027 requested a full tablet report, sending...
I20240502 14:23:11.808166 5592 ts_manager.cc:194] Registered new tserver with Master: 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015)
I20240502 14:23:11.808821 5592 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.66:39249
W20240502 14:23:12.087522 5912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:23:12.087781 5912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:23:12.087954 5912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:23:12.094300 5912 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:23:12.094410 5912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:23:12.094548 5912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.67
I20240502 14:23:12.102110 5912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.107.67
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.5.107.126:34027
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.67
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:23:12.103143 5912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:23:12.104620 5912 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:23:12.115312 5917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:23:12.115873 5918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:23:12.116027 5920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:23:12.116472 5912 server_base.cc:1034] running on GCE node
I20240502 14:23:12.116947 5912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:23:12.117710 5912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:23:12.118901 5912 hybrid_clock.cc:648] HybridClock initialized: now 1714659792118866 us; error 25 us; skew 500 ppm
I20240502 14:23:12.119131 5912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:23:12.121031 5912 webserver.cc:460] Webserver started at http://127.5.107.67:38965/ using document root <none> and password file <none>
I20240502 14:23:12.121691 5912 fs_manager.cc:359] Metadata directory not provided
I20240502 14:23:12.121780 5912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:23:12.122032 5912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:23:12.124140 5912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "181b466de0084fcbace0ca327fb1cbef"
format_stamp: "Formatted at 2024-05-02 14:23:12 on dist-test-slave-2cw1"
I20240502 14:23:12.124783 5912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "181b466de0084fcbace0ca327fb1cbef"
format_stamp: "Formatted at 2024-05-02 14:23:12 on dist-test-slave-2cw1"
I20240502 14:23:12.129155 5912 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20240502 14:23:12.132010 5926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:12.133248 5912 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:23:12.133369 5912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "181b466de0084fcbace0ca327fb1cbef"
format_stamp: "Formatted at 2024-05-02 14:23:12 on dist-test-slave-2cw1"
I20240502 14:23:12.133512 5912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:23:12.143898 5912 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:23:12.144528 5912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:23:12.144713 5912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:23:12.145641 5912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:23:12.146869 5912 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:23:12.146943 5912 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:12.147011 5912 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:23:12.147058 5912 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:12.175819 5912 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.67:41517
I20240502 14:23:12.175880 6038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.67:41517 every 8 connection(s)
I20240502 14:23:12.176954 5912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 14:23:12.185892 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 5912
I20240502 14:23:12.186122 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 14:23:12.188920 6039 heartbeater.cc:347] Connected to a master server at 127.5.107.126:34027
I20240502 14:23:12.189213 6039 heartbeater.cc:464] Registering TS with master...
I20240502 14:23:12.189491 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.68:0
--local_ip_for_outbound_sockets=127.5.107.68
--webserver_interface=127.5.107.68
--webserver_port=0
--tserver_master_addrs=127.5.107.126:34027
--builtin_ntp_servers=127.5.107.84:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 14:23:12.189966 6039 heartbeater.cc:511] Master 127.5.107.126:34027 requested a full tablet report, sending...
I20240502 14:23:12.191241 5592 ts_manager.cc:194] Registered new tserver with Master: 181b466de0084fcbace0ca327fb1cbef (127.5.107.67:41517)
I20240502 14:23:12.191994 5592 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.67:43277
I20240502 14:23:12.424294 5779 heartbeater.cc:503] Master 127.5.107.126:34027 was elected leader, sending a full tablet report...
W20240502 14:23:12.482470 6042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:23:12.482692 6042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:23:12.482795 6042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:23:12.489051 6042 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:23:12.489140 6042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:23:12.489284 6042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.68
I20240502 14:23:12.496838 6042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.5.107.68
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.5.107.126:34027
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.68
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:23:12.497859 6042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:23:12.499290 6042 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:23:12.509471 6048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:23:12.510038 6047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:23:12.510572 6050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:23:12.511139 6042 server_base.cc:1034] running on GCE node
I20240502 14:23:12.511628 6042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:23:12.512428 6042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:23:12.513603 6042 hybrid_clock.cc:648] HybridClock initialized: now 1714659792513560 us; error 40 us; skew 500 ppm
I20240502 14:23:12.513887 6042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:23:12.515751 6042 webserver.cc:460] Webserver started at http://127.5.107.68:45539/ using document root <none> and password file <none>
I20240502 14:23:12.516415 6042 fs_manager.cc:359] Metadata directory not provided
I20240502 14:23:12.516517 6042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:23:12.516783 6042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:23:12.519167 6042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "333acff728274715a9816c286921ddd0"
format_stamp: "Formatted at 2024-05-02 14:23:12 on dist-test-slave-2cw1"
I20240502 14:23:12.519896 6042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "333acff728274715a9816c286921ddd0"
format_stamp: "Formatted at 2024-05-02 14:23:12 on dist-test-slave-2cw1"
I20240502 14:23:12.524338 6042 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20240502 14:23:12.527334 6056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:12.528409 6042 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 14:23:12.528542 6042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "333acff728274715a9816c286921ddd0"
format_stamp: "Formatted at 2024-05-02 14:23:12 on dist-test-slave-2cw1"
I20240502 14:23:12.528704 6042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:23:12.554925 6042 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:23:12.555646 6042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:23:12.555897 6042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:23:12.556762 6042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:23:12.558039 6042 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:23:12.558125 6042 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:12.558219 6042 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:23:12.558280 6042 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:12.585876 6042 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.68:45913
I20240502 14:23:12.585955 6168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.68:45913 every 8 connection(s)
I20240502 14:23:12.587054 6042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:23:12.594712 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 6042
I20240502 14:23:12.594890 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 14:23:12.598166 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.69:0
--local_ip_for_outbound_sockets=127.5.107.69
--webserver_interface=127.5.107.69
--webserver_port=0
--tserver_master_addrs=127.5.107.126:34027
--builtin_ntp_servers=127.5.107.84:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 14:23:12.598282 6169 heartbeater.cc:347] Connected to a master server at 127.5.107.126:34027
I20240502 14:23:12.598548 6169 heartbeater.cc:464] Registering TS with master...
I20240502 14:23:12.599450 6169 heartbeater.cc:511] Master 127.5.107.126:34027 requested a full tablet report, sending...
I20240502 14:23:12.600777 5592 ts_manager.cc:194] Registered new tserver with Master: 333acff728274715a9816c286921ddd0 (127.5.107.68:45913)
I20240502 14:23:12.601622 5592 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.68:34213
I20240502 14:23:12.811012 5909 heartbeater.cc:503] Master 127.5.107.126:34027 was elected leader, sending a full tablet report...
W20240502 14:23:12.888396 6172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:23:12.888651 6172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:23:12.888788 6172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:23:12.895282 6172 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:23:12.895401 6172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:23:12.895551 6172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.69
I20240502 14:23:12.903220 6172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.5.107.69
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.5.107.126:34027
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.69
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:23:12.904328 6172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:23:12.905944 6172 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:23:12.916513 6177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:23:12.917088 6178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:23:12.917733 6180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:23:12.917879 6172 server_base.cc:1034] running on GCE node
I20240502 14:23:12.918426 6172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:23:12.919310 6172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:23:12.920507 6172 hybrid_clock.cc:648] HybridClock initialized: now 1714659792920446 us; error 56 us; skew 500 ppm
I20240502 14:23:12.920782 6172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:23:12.922948 6172 webserver.cc:460] Webserver started at http://127.5.107.69:34655/ using document root <none> and password file <none>
I20240502 14:23:12.923599 6172 fs_manager.cc:359] Metadata directory not provided
I20240502 14:23:12.923700 6172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:23:12.923985 6172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:23:12.926349 6172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "3416b273466c495298342b6e892015c2"
format_stamp: "Formatted at 2024-05-02 14:23:12 on dist-test-slave-2cw1"
I20240502 14:23:12.927011 6172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "3416b273466c495298342b6e892015c2"
format_stamp: "Formatted at 2024-05-02 14:23:12 on dist-test-slave-2cw1"
I20240502 14:23:12.931320 6172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:23:12.934242 6186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:12.935319 6172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:23:12.935467 6172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "3416b273466c495298342b6e892015c2"
format_stamp: "Formatted at 2024-05-02 14:23:12 on dist-test-slave-2cw1"
I20240502 14:23:12.935643 6172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:23:12.960413 6172 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:23:12.961081 6172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:23:12.961325 6172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:23:12.962100 6172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:23:12.963321 6172 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:23:12.963407 6172 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:12.963510 6172 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:23:12.963572 6172 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:12.991890 6172 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.69:37277
I20240502 14:23:12.991959 6298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.69:37277 every 8 connection(s)
I20240502 14:23:12.993023 6172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:23:13.002985 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 6172
I20240502 14:23:13.003165 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 14:23:13.003877 6299 heartbeater.cc:347] Connected to a master server at 127.5.107.126:34027
I20240502 14:23:13.004125 6299 heartbeater.cc:464] Registering TS with master...
I20240502 14:23:13.004947 6299 heartbeater.cc:511] Master 127.5.107.126:34027 requested a full tablet report, sending...
I20240502 14:23:13.006050 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.70:0
--local_ip_for_outbound_sockets=127.5.107.70
--webserver_interface=127.5.107.70
--webserver_port=0
--tserver_master_addrs=127.5.107.126:34027
--builtin_ntp_servers=127.5.107.84:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 14:23:13.006273 5592 ts_manager.cc:194] Registered new tserver with Master: 3416b273466c495298342b6e892015c2 (127.5.107.69:37277)
I20240502 14:23:13.006969 5592 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.69:42931
I20240502 14:23:13.194408 6039 heartbeater.cc:503] Master 127.5.107.126:34027 was elected leader, sending a full tablet report...
W20240502 14:23:13.295356 6302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:23:13.295631 6302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:23:13.295791 6302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:23:13.302328 6302 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:23:13.302459 6302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:23:13.302611 6302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.70
I20240502 14:23:13.310417 6302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.5.107.70
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.5.107.126:34027
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.70
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:23:13.311502 6302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:23:13.313019 6302 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:23:13.324014 6308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:23:13.324646 6310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:23:13.325218 6307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:23:13.325488 6302 server_base.cc:1034] running on GCE node
I20240502 14:23:13.326028 6302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:23:13.326933 6302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:23:13.328125 6302 hybrid_clock.cc:648] HybridClock initialized: now 1714659793328086 us; error 38 us; skew 500 ppm
I20240502 14:23:13.328392 6302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:23:13.330371 6302 webserver.cc:460] Webserver started at http://127.5.107.70:39573/ using document root <none> and password file <none>
I20240502 14:23:13.331022 6302 fs_manager.cc:359] Metadata directory not provided
I20240502 14:23:13.331130 6302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:23:13.331398 6302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:23:13.333824 6302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "806206bb049040bf8db48a7d8d3840c1"
format_stamp: "Formatted at 2024-05-02 14:23:13 on dist-test-slave-2cw1"
I20240502 14:23:13.334527 6302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "806206bb049040bf8db48a7d8d3840c1"
format_stamp: "Formatted at 2024-05-02 14:23:13 on dist-test-slave-2cw1"
I20240502 14:23:13.339190 6302 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:23:13.342119 6316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:13.343223 6302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:23:13.343355 6302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "806206bb049040bf8db48a7d8d3840c1"
format_stamp: "Formatted at 2024-05-02 14:23:13 on dist-test-slave-2cw1"
I20240502 14:23:13.343508 6302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:23:13.360610 6302 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:23:13.361378 6302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:23:13.361616 6302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:23:13.362485 6302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:23:13.363708 6302 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:23:13.363804 6302 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:13.363927 6302 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:23:13.364012 6302 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:13.391467 6302 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.70:40027
I20240502 14:23:13.391517 6428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.70:40027 every 8 connection(s)
I20240502 14:23:13.392597 6302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 14:23:13.400686 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 6302
I20240502 14:23:13.400844 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 14:23:13.403757 6429 heartbeater.cc:347] Connected to a master server at 127.5.107.126:34027
I20240502 14:23:13.404021 6429 heartbeater.cc:464] Registering TS with master...
I20240502 14:23:13.404443 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.71:0
--local_ip_for_outbound_sockets=127.5.107.71
--webserver_interface=127.5.107.71
--webserver_port=0
--tserver_master_addrs=127.5.107.126:34027
--builtin_ntp_servers=127.5.107.84:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 14:23:13.404891 6429 heartbeater.cc:511] Master 127.5.107.126:34027 requested a full tablet report, sending...
I20240502 14:23:13.406476 5592 ts_manager.cc:194] Registered new tserver with Master: 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70:40027)
I20240502 14:23:13.407328 5592 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.70:45651
I20240502 14:23:13.603868 6169 heartbeater.cc:503] Master 127.5.107.126:34027 was elected leader, sending a full tablet report...
W20240502 14:23:13.702275 6432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:23:13.702544 6432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:23:13.702684 6432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:23:13.709327 6432 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:23:13.709435 6432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:23:13.709568 6432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.71
I20240502 14:23:13.717602 6432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.5.107.71
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.5.107.126:34027
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.71
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:23:13.718672 6432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:23:13.720345 6432 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:23:13.731392 6440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:23:13.731844 6438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:23:13.731981 6437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:23:13.732501 6432 server_base.cc:1034] running on GCE node
I20240502 14:23:13.732964 6432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:23:13.733747 6432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:23:13.734922 6432 hybrid_clock.cc:648] HybridClock initialized: now 1714659793734881 us; error 36 us; skew 500 ppm
I20240502 14:23:13.735211 6432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:23:13.737346 6432 webserver.cc:460] Webserver started at http://127.5.107.71:35123/ using document root <none> and password file <none>
I20240502 14:23:13.737978 6432 fs_manager.cc:359] Metadata directory not provided
I20240502 14:23:13.738072 6432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:23:13.738404 6432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:23:13.740607 6432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "681fd3c49e914094bf28a7df155cffc9"
format_stamp: "Formatted at 2024-05-02 14:23:13 on dist-test-slave-2cw1"
I20240502 14:23:13.741341 6432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "681fd3c49e914094bf28a7df155cffc9"
format_stamp: "Formatted at 2024-05-02 14:23:13 on dist-test-slave-2cw1"
I20240502 14:23:13.745828 6432 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240502 14:23:13.748657 6446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:13.749747 6432 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:23:13.749886 6432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "681fd3c49e914094bf28a7df155cffc9"
format_stamp: "Formatted at 2024-05-02 14:23:13 on dist-test-slave-2cw1"
I20240502 14:23:13.750077 6432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:23:13.784701 6432 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:23:13.785549 6432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:23:13.785804 6432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:23:13.786660 6432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:23:13.787889 6432 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:23:13.787973 6432 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:13.788084 6432 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:23:13.788146 6432 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:13.818581 6432 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.71:42329
I20240502 14:23:13.818635 6558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.71:42329 every 8 connection(s)
I20240502 14:23:13.819703 6432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 14:23:13.829521 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 6432
I20240502 14:23:13.829705 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 14:23:13.830536 6559 heartbeater.cc:347] Connected to a master server at 127.5.107.126:34027
I20240502 14:23:13.830791 6559 heartbeater.cc:464] Registering TS with master...
I20240502 14:23:13.831537 6559 heartbeater.cc:511] Master 127.5.107.126:34027 requested a full tablet report, sending...
I20240502 14:23:13.832427 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.72:0
--local_ip_for_outbound_sockets=127.5.107.72
--webserver_interface=127.5.107.72
--webserver_port=0
--tserver_master_addrs=127.5.107.126:34027
--builtin_ntp_servers=127.5.107.84:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 14:23:13.833467 5592 ts_manager.cc:194] Registered new tserver with Master: 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71:42329)
I20240502 14:23:13.834113 5592 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.71:35279
I20240502 14:23:14.009071 6299 heartbeater.cc:503] Master 127.5.107.126:34027 was elected leader, sending a full tablet report...
W20240502 14:23:14.131690 6562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:23:14.131978 6562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:23:14.132115 6562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:23:14.139463 6562 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:23:14.139585 6562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:23:14.139780 6562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.72
I20240502 14:23:14.147861 6562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.5.107.72
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.5.107.126:34027
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.72
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:23:14.149044 6562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:23:14.150692 6562 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:23:14.161737 6567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:23:14.162292 6570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:23:14.162843 6568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:23:14.163182 6562 server_base.cc:1034] running on GCE node
I20240502 14:23:14.163625 6562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:23:14.164402 6562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:23:14.165589 6562 hybrid_clock.cc:648] HybridClock initialized: now 1714659794165567 us; error 34 us; skew 500 ppm
I20240502 14:23:14.165869 6562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:23:14.168116 6562 webserver.cc:460] Webserver started at http://127.5.107.72:36735/ using document root <none> and password file <none>
I20240502 14:23:14.168762 6562 fs_manager.cc:359] Metadata directory not provided
I20240502 14:23:14.168874 6562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:23:14.169239 6562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:23:14.171680 6562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "83b0e3c0ba724317832f8505a1ffb66e"
format_stamp: "Formatted at 2024-05-02 14:23:14 on dist-test-slave-2cw1"
I20240502 14:23:14.172371 6562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "83b0e3c0ba724317832f8505a1ffb66e"
format_stamp: "Formatted at 2024-05-02 14:23:14 on dist-test-slave-2cw1"
I20240502 14:23:14.177321 6562 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20240502 14:23:14.180263 6576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:14.181545 6562 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:23:14.181699 6562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "83b0e3c0ba724317832f8505a1ffb66e"
format_stamp: "Formatted at 2024-05-02 14:23:14 on dist-test-slave-2cw1"
I20240502 14:23:14.181876 6562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:23:14.197093 6562 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:23:14.197860 6562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:23:14.198091 6562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:23:14.198956 6562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:23:14.200284 6562 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:23:14.200371 6562 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:14.200485 6562 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:23:14.200551 6562 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:14.230111 6562 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.72:35323
I20240502 14:23:14.230167 6688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.72:35323 every 8 connection(s)
I20240502 14:23:14.231277 6562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 14:23:14.241508 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 6562
I20240502 14:23:14.241681 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 14:23:14.242980 6689 heartbeater.cc:347] Connected to a master server at 127.5.107.126:34027
I20240502 14:23:14.243245 6689 heartbeater.cc:464] Registering TS with master...
I20240502 14:23:14.243996 6689 heartbeater.cc:511] Master 127.5.107.126:34027 requested a full tablet report, sending...
I20240502 14:23:14.245476 5591 ts_manager.cc:194] Registered new tserver with Master: 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72:35323)
I20240502 14:23:14.246487 5591 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.72:54795
I20240502 14:23:14.257983 5549 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 14:23:14.277493 5549 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 14:23:14.277650 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 6172
I20240502 14:23:14.285001 6294 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:23:14.362210 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 6172
I20240502 14:23:14.378684 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 6302
I20240502 14:23:14.386824 6424 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:23:14.461009 6429 heartbeater.cc:503] Master 127.5.107.126:34027 was elected leader, sending a full tablet report...
I20240502 14:23:14.462270 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 6302
W20240502 14:23:14.477331 5578 connection.cc:535] server connection from 127.5.107.70:45651 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:23:14.477641 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 6432
I20240502 14:23:14.485244 6554 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:23:14.558619 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 6432
I20240502 14:23:14.573335 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 6562
I20240502 14:23:14.581689 6684 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:23:14.655401 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 6562
I20240502 14:23:18.424036 5591 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54758:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:23:18.458072 6103 tablet_service.cc:1467] Processing CreateTablet for tablet 7d944a63fdf24ecd8fadbad3b620af0a (DEFAULT_TABLE table=rebalance_test_table_0 [id=7a7408907bfc4fd2bd39cf4e8b4c97fb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:18.459151 5713 tablet_service.cc:1467] Processing CreateTablet for tablet 228133f033a44f59af1259cc1a1e9d55 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7a7408907bfc4fd2bd39cf4e8b4c97fb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:18.460214 5713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 228133f033a44f59af1259cc1a1e9d55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:18.460137 5844 tablet_service.cc:1467] Processing CreateTablet for tablet 228133f033a44f59af1259cc1a1e9d55 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7a7408907bfc4fd2bd39cf4e8b4c97fb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:18.460621 5843 tablet_service.cc:1467] Processing CreateTablet for tablet 7d944a63fdf24ecd8fadbad3b620af0a (DEFAULT_TABLE table=rebalance_test_table_0 [id=7a7408907bfc4fd2bd39cf4e8b4c97fb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:18.461436 5843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d944a63fdf24ecd8fadbad3b620af0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:18.461375 5712 tablet_service.cc:1467] Processing CreateTablet for tablet 7d944a63fdf24ecd8fadbad3b620af0a (DEFAULT_TABLE table=rebalance_test_table_0 [id=7a7408907bfc4fd2bd39cf4e8b4c97fb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:18.461612 5974 tablet_service.cc:1467] Processing CreateTablet for tablet aca9200e5c264c27833d7a4a9122c4d9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7a7408907bfc4fd2bd39cf4e8b4c97fb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:18.462119 5973 tablet_service.cc:1467] Processing CreateTablet for tablet 228133f033a44f59af1259cc1a1e9d55 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7a7408907bfc4fd2bd39cf4e8b4c97fb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:18.462510 5712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d944a63fdf24ecd8fadbad3b620af0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:18.463032 5973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 228133f033a44f59af1259cc1a1e9d55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:18.463459 5974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aca9200e5c264c27833d7a4a9122c4d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:18.464505 5844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 228133f033a44f59af1259cc1a1e9d55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:18.465283 6103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d944a63fdf24ecd8fadbad3b620af0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:18.454841 5714 tablet_service.cc:1467] Processing CreateTablet for tablet aca9200e5c264c27833d7a4a9122c4d9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7a7408907bfc4fd2bd39cf4e8b4c97fb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:18.466917 5714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aca9200e5c264c27833d7a4a9122c4d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:18.464136 6104 tablet_service.cc:1467] Processing CreateTablet for tablet aca9200e5c264c27833d7a4a9122c4d9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7a7408907bfc4fd2bd39cf4e8b4c97fb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:18.471694 6104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aca9200e5c264c27833d7a4a9122c4d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:18.480729 6718 tablet_bootstrap.cc:492] T 7d944a63fdf24ecd8fadbad3b620af0a P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap starting.
I20240502 14:23:18.483367 6719 tablet_bootstrap.cc:492] T 228133f033a44f59af1259cc1a1e9d55 P 181b466de0084fcbace0ca327fb1cbef: Bootstrap starting.
I20240502 14:23:18.485080 6718 tablet_bootstrap.cc:654] T 7d944a63fdf24ecd8fadbad3b620af0a P 1dbf482780eb47e8933983e4f0fd0444: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:18.486526 6718 log.cc:824] T 7d944a63fdf24ecd8fadbad3b620af0a P 1dbf482780eb47e8933983e4f0fd0444: Log is configured to *not* fsync() on all Append() calls
I20240502 14:23:18.487390 6719 tablet_bootstrap.cc:654] T 228133f033a44f59af1259cc1a1e9d55 P 181b466de0084fcbace0ca327fb1cbef: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:18.488847 6719 log.cc:824] T 228133f033a44f59af1259cc1a1e9d55 P 181b466de0084fcbace0ca327fb1cbef: Log is configured to *not* fsync() on all Append() calls
I20240502 14:23:18.490098 6720 tablet_bootstrap.cc:492] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0: Bootstrap starting.
I20240502 14:23:18.491966 6723 tablet_bootstrap.cc:492] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081: Bootstrap starting.
I20240502 14:23:18.494299 6718 tablet_bootstrap.cc:492] T 7d944a63fdf24ecd8fadbad3b620af0a P 1dbf482780eb47e8933983e4f0fd0444: No bootstrap required, opened a new log
I20240502 14:23:18.494529 6718 ts_tablet_manager.cc:1367] T 7d944a63fdf24ecd8fadbad3b620af0a P 1dbf482780eb47e8933983e4f0fd0444: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20240502 14:23:18.494868 6723 tablet_bootstrap.cc:654] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:18.496006 6723 log.cc:824] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081: Log is configured to *not* fsync() on all Append() calls
I20240502 14:23:18.496330 6719 tablet_bootstrap.cc:492] T 228133f033a44f59af1259cc1a1e9d55 P 181b466de0084fcbace0ca327fb1cbef: No bootstrap required, opened a new log
I20240502 14:23:18.496546 6720 tablet_bootstrap.cc:654] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:18.496650 6719 ts_tablet_manager.cc:1367] T 228133f033a44f59af1259cc1a1e9d55 P 181b466de0084fcbace0ca327fb1cbef: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20240502 14:23:18.497934 6720 log.cc:824] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0: Log is configured to *not* fsync() on all Append() calls
I20240502 14:23:18.502920 6718 raft_consensus.cc:348] T 7d944a63fdf24ecd8fadbad3b620af0a P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:18.503396 6718 raft_consensus.cc:374] T 7d944a63fdf24ecd8fadbad3b620af0a P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:18.503532 6718 raft_consensus.cc:729] T 7d944a63fdf24ecd8fadbad3b620af0a P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbf482780eb47e8933983e4f0fd0444, State: Initialized, Role: FOLLOWER
I20240502 14:23:18.504300 6718 consensus_queue.cc:260] T 7d944a63fdf24ecd8fadbad3b620af0a P 1dbf482780eb47e8933983e4f0fd0444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:18.505928 6719 raft_consensus.cc:348] T 228133f033a44f59af1259cc1a1e9d55 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:18.506680 6719 raft_consensus.cc:374] T 228133f033a44f59af1259cc1a1e9d55 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:18.507802 6719 raft_consensus.cc:729] T 228133f033a44f59af1259cc1a1e9d55 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 181b466de0084fcbace0ca327fb1cbef, State: Initialized, Role: FOLLOWER
I20240502 14:23:18.508483 6718 ts_tablet_manager.cc:1398] T 7d944a63fdf24ecd8fadbad3b620af0a P 1dbf482780eb47e8933983e4f0fd0444: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20240502 14:23:18.508477 6719 consensus_queue.cc:260] T 228133f033a44f59af1259cc1a1e9d55 P 181b466de0084fcbace0ca327fb1cbef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:18.509083 6723 tablet_bootstrap.cc:492] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081: No bootstrap required, opened a new log
I20240502 14:23:18.509256 6720 tablet_bootstrap.cc:492] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0: No bootstrap required, opened a new log
I20240502 14:23:18.509313 6723 ts_tablet_manager.cc:1367] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081: Time spent bootstrapping tablet: real 0.018s user 0.017s sys 0.000s
I20240502 14:23:18.509541 6720 ts_tablet_manager.cc:1367] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0: Time spent bootstrapping tablet: real 0.020s user 0.016s sys 0.000s
I20240502 14:23:18.515386 6718 tablet_bootstrap.cc:492] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap starting.
I20240502 14:23:18.517000 6718 tablet_bootstrap.cc:654] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:18.514828 6723 raft_consensus.cc:348] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:18.517689 6723 raft_consensus.cc:374] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:18.517777 6723 raft_consensus.cc:729] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e62d4e5f8a4316bc1bcb8c18458081, State: Initialized, Role: FOLLOWER
I20240502 14:23:18.519888 6720 raft_consensus.cc:348] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:18.520015 6723 consensus_queue.cc:260] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:18.520309 6720 raft_consensus.cc:374] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:18.520421 6720 raft_consensus.cc:729] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 333acff728274715a9816c286921ddd0, State: Initialized, Role: FOLLOWER
I20240502 14:23:18.521365 6720 consensus_queue.cc:260] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:18.523890 6719 ts_tablet_manager.cc:1398] T 228133f033a44f59af1259cc1a1e9d55 P 181b466de0084fcbace0ca327fb1cbef: Time spent starting tablet: real 0.027s user 0.017s sys 0.006s
I20240502 14:23:18.524252 6719 tablet_bootstrap.cc:492] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef: Bootstrap starting.
I20240502 14:23:18.524859 6718 tablet_bootstrap.cc:492] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444: No bootstrap required, opened a new log
I20240502 14:23:18.525086 6718 ts_tablet_manager.cc:1367] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20240502 14:23:18.525795 6719 tablet_bootstrap.cc:654] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:18.526404 6718 raft_consensus.cc:348] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:18.526762 6718 raft_consensus.cc:374] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:18.526909 6718 raft_consensus.cc:729] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbf482780eb47e8933983e4f0fd0444, State: Initialized, Role: FOLLOWER
I20240502 14:23:18.527237 6718 consensus_queue.cc:260] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:18.528761 6718 ts_tablet_manager.cc:1398] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240502 14:23:18.529125 6718 tablet_bootstrap.cc:492] T 228133f033a44f59af1259cc1a1e9d55 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap starting.
I20240502 14:23:18.532322 6718 tablet_bootstrap.cc:654] T 228133f033a44f59af1259cc1a1e9d55 P 1dbf482780eb47e8933983e4f0fd0444: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:18.534130 6719 tablet_bootstrap.cc:492] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef: No bootstrap required, opened a new log
I20240502 14:23:18.534281 6719 ts_tablet_manager.cc:1367] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240502 14:23:18.534895 6723 ts_tablet_manager.cc:1398] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081: Time spent starting tablet: real 0.025s user 0.013s sys 0.007s
I20240502 14:23:18.535315 6723 tablet_bootstrap.cc:492] T 7d944a63fdf24ecd8fadbad3b620af0a P 83e62d4e5f8a4316bc1bcb8c18458081: Bootstrap starting.
I20240502 14:23:18.535295 6726 raft_consensus.cc:482] T 7d944a63fdf24ecd8fadbad3b620af0a P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:23:18.535224 6719 raft_consensus.cc:348] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:18.535439 6719 raft_consensus.cc:374] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:18.535542 6719 raft_consensus.cc:729] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 181b466de0084fcbace0ca327fb1cbef, State: Initialized, Role: FOLLOWER
I20240502 14:23:18.535528 6726 raft_consensus.cc:504] T 7d944a63fdf24ecd8fadbad3b620af0a P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:18.535737 6719 consensus_queue.cc:260] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:18.536253 6719 ts_tablet_manager.cc:1398] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:23:18.536969 6723 tablet_bootstrap.cc:654] T 7d944a63fdf24ecd8fadbad3b620af0a P 83e62d4e5f8a4316bc1bcb8c18458081: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:18.537410 6726 leader_election.cc:288] T 7d944a63fdf24ecd8fadbad3b620af0a P 1dbf482780eb47e8933983e4f0fd0444 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 333acff728274715a9816c286921ddd0 (127.5.107.68:45913), 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015)
I20240502 14:23:18.539954 6720 ts_tablet_manager.cc:1398] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0: Time spent starting tablet: real 0.030s user 0.008s sys 0.011s
I20240502 14:23:18.541455 6720 tablet_bootstrap.cc:492] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0: Bootstrap starting.
I20240502 14:23:18.542954 6720 tablet_bootstrap.cc:654] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0: Neither blocks nor log segments found. Creating new log.
W20240502 14:23:18.549842 5910 tablet.cc:2369] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:23:18.555740 6718 tablet_bootstrap.cc:492] T 228133f033a44f59af1259cc1a1e9d55 P 1dbf482780eb47e8933983e4f0fd0444: No bootstrap required, opened a new log
I20240502 14:23:18.555976 6718 ts_tablet_manager.cc:1367] T 228133f033a44f59af1259cc1a1e9d55 P 1dbf482780eb47e8933983e4f0fd0444: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.001s
I20240502 14:23:18.556948 6718 raft_consensus.cc:348] T 228133f033a44f59af1259cc1a1e9d55 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:18.557255 6718 raft_consensus.cc:374] T 228133f033a44f59af1259cc1a1e9d55 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:18.557343 6718 raft_consensus.cc:729] T 228133f033a44f59af1259cc1a1e9d55 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbf482780eb47e8933983e4f0fd0444, State: Initialized, Role: FOLLOWER
I20240502 14:23:18.557615 6718 consensus_queue.cc:260] T 228133f033a44f59af1259cc1a1e9d55 P 1dbf482780eb47e8933983e4f0fd0444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:18.558346 6718 ts_tablet_manager.cc:1398] T 228133f033a44f59af1259cc1a1e9d55 P 1dbf482780eb47e8933983e4f0fd0444: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:23:18.562611 5864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d944a63fdf24ecd8fadbad3b620af0a" candidate_uuid: "1dbf482780eb47e8933983e4f0fd0444" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" is_pre_election: true
W20240502 14:23:18.563818 5668 leader_election.cc:341] T 7d944a63fdf24ecd8fadbad3b620af0a P 1dbf482780eb47e8933983e4f0fd0444 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:23:18.564539 6723 tablet_bootstrap.cc:492] T 7d944a63fdf24ecd8fadbad3b620af0a P 83e62d4e5f8a4316bc1bcb8c18458081: No bootstrap required, opened a new log
I20240502 14:23:18.564343 6124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d944a63fdf24ecd8fadbad3b620af0a" candidate_uuid: "1dbf482780eb47e8933983e4f0fd0444" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "333acff728274715a9816c286921ddd0" is_pre_election: true
I20240502 14:23:18.564662 6723 ts_tablet_manager.cc:1367] T 7d944a63fdf24ecd8fadbad3b620af0a P 83e62d4e5f8a4316bc1bcb8c18458081: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.016s
W20240502 14:23:18.565451 5669 leader_election.cc:341] T 7d944a63fdf24ecd8fadbad3b620af0a P 1dbf482780eb47e8933983e4f0fd0444 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 333acff728274715a9816c286921ddd0 (127.5.107.68:45913): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:23:18.565451 6723 raft_consensus.cc:348] T 7d944a63fdf24ecd8fadbad3b620af0a P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:18.565662 6723 raft_consensus.cc:374] T 7d944a63fdf24ecd8fadbad3b620af0a P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:18.565624 5669 leader_election.cc:302] T 7d944a63fdf24ecd8fadbad3b620af0a P 1dbf482780eb47e8933983e4f0fd0444 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1dbf482780eb47e8933983e4f0fd0444; no voters: 333acff728274715a9816c286921ddd0, 83e62d4e5f8a4316bc1bcb8c18458081
I20240502 14:23:18.565759 6723 raft_consensus.cc:729] T 7d944a63fdf24ecd8fadbad3b620af0a P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e62d4e5f8a4316bc1bcb8c18458081, State: Initialized, Role: FOLLOWER
I20240502 14:23:18.565979 6723 consensus_queue.cc:260] T 7d944a63fdf24ecd8fadbad3b620af0a P 83e62d4e5f8a4316bc1bcb8c18458081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:18.566223 6720 tablet_bootstrap.cc:492] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0: No bootstrap required, opened a new log
I20240502 14:23:18.566303 6726 raft_consensus.cc:2726] T 7d944a63fdf24ecd8fadbad3b620af0a P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:23:18.566522 6723 ts_tablet_manager.cc:1398] T 7d944a63fdf24ecd8fadbad3b620af0a P 83e62d4e5f8a4316bc1bcb8c18458081: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:23:18.566756 6720 ts_tablet_manager.cc:1367] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.016s
I20240502 14:23:18.567494 6720 raft_consensus.cc:348] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:18.567632 6720 raft_consensus.cc:374] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:18.567713 6720 raft_consensus.cc:729] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 333acff728274715a9816c286921ddd0, State: Initialized, Role: FOLLOWER
I20240502 14:23:18.567915 6720 consensus_queue.cc:260] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:18.568290 6720 ts_tablet_manager.cc:1398] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240502 14:23:18.592501 6170 tablet.cc:2369] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:23:18.609076 6729 raft_consensus.cc:482] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:23:18.609314 6729 raft_consensus.cc:504] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:18.610780 6729 leader_election.cc:288] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831), 181b466de0084fcbace0ca327fb1cbef (127.5.107.67:41517)
I20240502 14:23:18.618880 5734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228133f033a44f59af1259cc1a1e9d55" candidate_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dbf482780eb47e8933983e4f0fd0444" is_pre_election: true
I20240502 14:23:18.618877 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228133f033a44f59af1259cc1a1e9d55" candidate_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "181b466de0084fcbace0ca327fb1cbef" is_pre_election: true
I20240502 14:23:18.619256 5734 raft_consensus.cc:2454] T 228133f033a44f59af1259cc1a1e9d55 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e62d4e5f8a4316bc1bcb8c18458081 in term 0.
I20240502 14:23:18.619321 5994 raft_consensus.cc:2454] T 228133f033a44f59af1259cc1a1e9d55 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e62d4e5f8a4316bc1bcb8c18458081 in term 0.
I20240502 14:23:18.619796 5799 leader_election.cc:302] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 181b466de0084fcbace0ca327fb1cbef, 83e62d4e5f8a4316bc1bcb8c18458081; no voters:
I20240502 14:23:18.620147 6729 raft_consensus.cc:2781] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:23:18.620270 6729 raft_consensus.cc:482] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:23:18.620347 6729 raft_consensus.cc:3037] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:18.622208 6729 raft_consensus.cc:504] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:18.622687 6729 leader_election.cc:288] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 [CANDIDATE]: Term 1 election: Requested vote from peers 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831), 181b466de0084fcbace0ca327fb1cbef (127.5.107.67:41517)
I20240502 14:23:18.623039 5734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228133f033a44f59af1259cc1a1e9d55" candidate_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dbf482780eb47e8933983e4f0fd0444"
I20240502 14:23:18.623083 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228133f033a44f59af1259cc1a1e9d55" candidate_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "181b466de0084fcbace0ca327fb1cbef"
I20240502 14:23:18.623180 5734 raft_consensus.cc:3037] T 228133f033a44f59af1259cc1a1e9d55 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:18.623240 5994 raft_consensus.cc:3037] T 228133f033a44f59af1259cc1a1e9d55 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:18.624927 5734 raft_consensus.cc:2454] T 228133f033a44f59af1259cc1a1e9d55 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e62d4e5f8a4316bc1bcb8c18458081 in term 1.
I20240502 14:23:18.624953 5994 raft_consensus.cc:2454] T 228133f033a44f59af1259cc1a1e9d55 P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e62d4e5f8a4316bc1bcb8c18458081 in term 1.
I20240502 14:23:18.625269 5797 leader_election.cc:302] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dbf482780eb47e8933983e4f0fd0444, 83e62d4e5f8a4316bc1bcb8c18458081; no voters:
I20240502 14:23:18.625526 6729 raft_consensus.cc:2781] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:23:18.625932 6729 raft_consensus.cc:686] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 LEADER]: Becoming Leader. State: Replica: 83e62d4e5f8a4316bc1bcb8c18458081, State: Running, Role: LEADER
I20240502 14:23:18.626395 6729 consensus_queue.cc:237] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:18.630432 5591 catalog_manager.cc:5484] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66). New cstate: current_term: 1 leader_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } health_report { overall_health: UNKNOWN } } }
I20240502 14:23:18.650597 6730 raft_consensus.cc:482] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:23:18.650790 6730 raft_consensus.cc:504] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:18.652114 6730 leader_election.cc:288] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831), 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015)
I20240502 14:23:18.659734 5734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d944a63fdf24ecd8fadbad3b620af0a" candidate_uuid: "333acff728274715a9816c286921ddd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dbf482780eb47e8933983e4f0fd0444" is_pre_election: true
I20240502 14:23:18.659734 5864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d944a63fdf24ecd8fadbad3b620af0a" candidate_uuid: "333acff728274715a9816c286921ddd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" is_pre_election: true
I20240502 14:23:18.659955 5734 raft_consensus.cc:2454] T 7d944a63fdf24ecd8fadbad3b620af0a P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 333acff728274715a9816c286921ddd0 in term 0.
I20240502 14:23:18.659967 5864 raft_consensus.cc:2454] T 7d944a63fdf24ecd8fadbad3b620af0a P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 333acff728274715a9816c286921ddd0 in term 0.
I20240502 14:23:18.660369 6058 leader_election.cc:302] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 333acff728274715a9816c286921ddd0, 83e62d4e5f8a4316bc1bcb8c18458081; no voters:
I20240502 14:23:18.660669 6730 raft_consensus.cc:2781] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:23:18.660800 6730 raft_consensus.cc:482] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:23:18.660912 6730 raft_consensus.cc:3037] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Advancing to term 1
W20240502 14:23:18.662161 5780 tablet.cc:2369] T 228133f033a44f59af1259cc1a1e9d55 P 1dbf482780eb47e8933983e4f0fd0444: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:23:18.663035 6730 raft_consensus.cc:504] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:18.663556 6730 leader_election.cc:288] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 1 election: Requested vote from peers 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831), 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015)
I20240502 14:23:18.663892 5734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d944a63fdf24ecd8fadbad3b620af0a" candidate_uuid: "333acff728274715a9816c286921ddd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dbf482780eb47e8933983e4f0fd0444"
I20240502 14:23:18.663933 5864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d944a63fdf24ecd8fadbad3b620af0a" candidate_uuid: "333acff728274715a9816c286921ddd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081"
I20240502 14:23:18.664057 5734 raft_consensus.cc:3037] T 7d944a63fdf24ecd8fadbad3b620af0a P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:18.664098 5864 raft_consensus.cc:3037] T 7d944a63fdf24ecd8fadbad3b620af0a P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:18.665692 5734 raft_consensus.cc:2454] T 7d944a63fdf24ecd8fadbad3b620af0a P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 333acff728274715a9816c286921ddd0 in term 1.
I20240502 14:23:18.665797 5864 raft_consensus.cc:2454] T 7d944a63fdf24ecd8fadbad3b620af0a P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 333acff728274715a9816c286921ddd0 in term 1.
I20240502 14:23:18.666056 6057 leader_election.cc:302] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dbf482780eb47e8933983e4f0fd0444, 333acff728274715a9816c286921ddd0; no voters:
I20240502 14:23:18.666359 6730 raft_consensus.cc:2781] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:23:18.666718 6730 raft_consensus.cc:686] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [term 1 LEADER]: Becoming Leader. State: Replica: 333acff728274715a9816c286921ddd0, State: Running, Role: LEADER
I20240502 14:23:18.667178 6730 consensus_queue.cc:237] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:18.670871 5591 catalog_manager.cc:5484] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 333acff728274715a9816c286921ddd0 (127.5.107.68). New cstate: current_term: 1 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } health_report { overall_health: UNKNOWN } } }
W20240502 14:23:18.682708 6040 tablet.cc:2369] T 228133f033a44f59af1259cc1a1e9d55 P 181b466de0084fcbace0ca327fb1cbef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:23:18.694275 6726 raft_consensus.cc:482] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:23:18.694413 6726 raft_consensus.cc:504] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:18.694970 6726 leader_election.cc:288] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 333acff728274715a9816c286921ddd0 (127.5.107.68:45913), 181b466de0084fcbace0ca327fb1cbef (127.5.107.67:41517)
I20240502 14:23:18.695423 6124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca9200e5c264c27833d7a4a9122c4d9" candidate_uuid: "1dbf482780eb47e8933983e4f0fd0444" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "333acff728274715a9816c286921ddd0" is_pre_election: true
I20240502 14:23:18.695693 6124 raft_consensus.cc:2454] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dbf482780eb47e8933983e4f0fd0444 in term 0.
I20240502 14:23:18.696159 5669 leader_election.cc:302] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dbf482780eb47e8933983e4f0fd0444, 333acff728274715a9816c286921ddd0; no voters:
I20240502 14:23:18.696415 6726 raft_consensus.cc:2781] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:23:18.696547 6726 raft_consensus.cc:482] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:23:18.696645 6726 raft_consensus.cc:3037] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:18.698864 6726 raft_consensus.cc:504] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:18.699470 6726 leader_election.cc:288] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444 [CANDIDATE]: Term 1 election: Requested vote from peers 333acff728274715a9816c286921ddd0 (127.5.107.68:45913), 181b466de0084fcbace0ca327fb1cbef (127.5.107.67:41517)
I20240502 14:23:18.699859 6124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca9200e5c264c27833d7a4a9122c4d9" candidate_uuid: "1dbf482780eb47e8933983e4f0fd0444" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "333acff728274715a9816c286921ddd0"
I20240502 14:23:18.700037 6124 raft_consensus.cc:3037] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:18.700758 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca9200e5c264c27833d7a4a9122c4d9" candidate_uuid: "1dbf482780eb47e8933983e4f0fd0444" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "181b466de0084fcbace0ca327fb1cbef" is_pre_election: true
I20240502 14:23:18.700831 5993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca9200e5c264c27833d7a4a9122c4d9" candidate_uuid: "1dbf482780eb47e8933983e4f0fd0444" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "181b466de0084fcbace0ca327fb1cbef"
I20240502 14:23:18.701068 5994 raft_consensus.cc:2454] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dbf482780eb47e8933983e4f0fd0444 in term 0.
I20240502 14:23:18.702426 6124 raft_consensus.cc:2454] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dbf482780eb47e8933983e4f0fd0444 in term 1.
I20240502 14:23:18.702720 5669 leader_election.cc:302] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1dbf482780eb47e8933983e4f0fd0444, 333acff728274715a9816c286921ddd0; no voters: 181b466de0084fcbace0ca327fb1cbef
I20240502 14:23:18.702960 6726 raft_consensus.cc:2781] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:23:18.703315 6726 raft_consensus.cc:686] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 LEADER]: Becoming Leader. State: Replica: 1dbf482780eb47e8933983e4f0fd0444, State: Running, Role: LEADER
I20240502 14:23:18.703768 6726 consensus_queue.cc:237] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:18.707278 5591 catalog_manager.cc:5484] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65). New cstate: current_term: 1 leader_uuid: "1dbf482780eb47e8933983e4f0fd0444" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } health_report { overall_health: UNKNOWN } } }
I20240502 14:23:19.007992 6729 consensus_queue.cc:1035] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:23:19.015302 6751 consensus_queue.cc:1035] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:23:19.023993 6747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:23:19.024228 6747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:23:19.030660 6747 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 14:23:19.100867 5993 raft_consensus.cc:3037] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:19.104506 6730 consensus_queue.cc:1035] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:23:19.105440 6726 consensus_queue.cc:1035] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444 [LEADER]: Connected to new peer: Peer: permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:23:19.120430 6730 consensus_queue.cc:1035] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:23:19.145411 6745 consensus_queue.cc:1035] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444 [LEADER]: Connected to new peer: Peer: permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 241.794 ms
time per row: 0.120897 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:23:20.244844 5589 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54758:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:23:20.252737 5973 tablet_service.cc:1467] Processing CreateTablet for tablet 08d82302438544e4925a4f511f13fb7e (DEFAULT_TABLE table=rebalance_test_table_1 [id=a3f5812bc1aa47c1990b9a769141e329]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:20.253080 6104 tablet_service.cc:1467] Processing CreateTablet for tablet 08d82302438544e4925a4f511f13fb7e (DEFAULT_TABLE table=rebalance_test_table_1 [id=a3f5812bc1aa47c1990b9a769141e329]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:20.253247 5973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08d82302438544e4925a4f511f13fb7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:20.253535 6104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08d82302438544e4925a4f511f13fb7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:20.253463 5844 tablet_service.cc:1467] Processing CreateTablet for tablet 08d82302438544e4925a4f511f13fb7e (DEFAULT_TABLE table=rebalance_test_table_1 [id=a3f5812bc1aa47c1990b9a769141e329]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:20.253849 5844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08d82302438544e4925a4f511f13fb7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:20.257865 6810 tablet_bootstrap.cc:492] T 08d82302438544e4925a4f511f13fb7e P 83e62d4e5f8a4316bc1bcb8c18458081: Bootstrap starting.
I20240502 14:23:20.258309 5843 tablet_service.cc:1467] Processing CreateTablet for tablet 81cfc8c8608b4cb0b7c303a910791d4b (DEFAULT_TABLE table=rebalance_test_table_1 [id=a3f5812bc1aa47c1990b9a769141e329]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:20.258644 5843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81cfc8c8608b4cb0b7c303a910791d4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:20.259420 6810 tablet_bootstrap.cc:654] T 08d82302438544e4925a4f511f13fb7e P 83e62d4e5f8a4316bc1bcb8c18458081: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:20.261679 5842 tablet_service.cc:1467] Processing CreateTablet for tablet 3128a2a3296e4940bfdf669fecfd7877 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a3f5812bc1aa47c1990b9a769141e329]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:20.262195 5842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3128a2a3296e4940bfdf669fecfd7877. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:20.266142 5712 tablet_service.cc:1467] Processing CreateTablet for tablet 3128a2a3296e4940bfdf669fecfd7877 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a3f5812bc1aa47c1990b9a769141e329]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:20.266652 5712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3128a2a3296e4940bfdf669fecfd7877. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:20.267555 6810 tablet_bootstrap.cc:492] T 08d82302438544e4925a4f511f13fb7e P 83e62d4e5f8a4316bc1bcb8c18458081: No bootstrap required, opened a new log
I20240502 14:23:20.267664 6810 ts_tablet_manager.cc:1367] T 08d82302438544e4925a4f511f13fb7e P 83e62d4e5f8a4316bc1bcb8c18458081: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20240502 14:23:20.268033 6811 tablet_bootstrap.cc:492] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0: Bootstrap starting.
I20240502 14:23:20.268143 6812 tablet_bootstrap.cc:492] T 08d82302438544e4925a4f511f13fb7e P 181b466de0084fcbace0ca327fb1cbef: Bootstrap starting.
I20240502 14:23:20.268327 6810 raft_consensus.cc:348] T 08d82302438544e4925a4f511f13fb7e P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:20.268496 6810 raft_consensus.cc:374] T 08d82302438544e4925a4f511f13fb7e P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:20.268565 6810 raft_consensus.cc:729] T 08d82302438544e4925a4f511f13fb7e P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e62d4e5f8a4316bc1bcb8c18458081, State: Initialized, Role: FOLLOWER
I20240502 14:23:20.268806 6810 consensus_queue.cc:260] T 08d82302438544e4925a4f511f13fb7e P 83e62d4e5f8a4316bc1bcb8c18458081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:20.269412 6810 ts_tablet_manager.cc:1398] T 08d82302438544e4925a4f511f13fb7e P 83e62d4e5f8a4316bc1bcb8c18458081: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:23:20.269479 6811 tablet_bootstrap.cc:654] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:20.270557 5972 tablet_service.cc:1467] Processing CreateTablet for tablet 3128a2a3296e4940bfdf669fecfd7877 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a3f5812bc1aa47c1990b9a769141e329]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:20.270908 5972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3128a2a3296e4940bfdf669fecfd7877. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:20.270579 5974 tablet_service.cc:1467] Processing CreateTablet for tablet 81cfc8c8608b4cb0b7c303a910791d4b (DEFAULT_TABLE table=rebalance_test_table_1 [id=a3f5812bc1aa47c1990b9a769141e329]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:20.272516 5974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81cfc8c8608b4cb0b7c303a910791d4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:20.276007 6812 tablet_bootstrap.cc:654] T 08d82302438544e4925a4f511f13fb7e P 181b466de0084fcbace0ca327fb1cbef: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:20.276094 6810 tablet_bootstrap.cc:492] T 3128a2a3296e4940bfdf669fecfd7877 P 83e62d4e5f8a4316bc1bcb8c18458081: Bootstrap starting.
I20240502 14:23:20.277534 6810 tablet_bootstrap.cc:654] T 3128a2a3296e4940bfdf669fecfd7877 P 83e62d4e5f8a4316bc1bcb8c18458081: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:20.278120 6811 tablet_bootstrap.cc:492] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0: No bootstrap required, opened a new log
I20240502 14:23:20.278259 6811 ts_tablet_manager.cc:1367] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20240502 14:23:20.278304 6812 tablet_bootstrap.cc:492] T 08d82302438544e4925a4f511f13fb7e P 181b466de0084fcbace0ca327fb1cbef: No bootstrap required, opened a new log
I20240502 14:23:20.278417 6812 ts_tablet_manager.cc:1367] T 08d82302438544e4925a4f511f13fb7e P 181b466de0084fcbace0ca327fb1cbef: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240502 14:23:20.278966 6811 raft_consensus.cc:348] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:20.279039 6812 raft_consensus.cc:348] T 08d82302438544e4925a4f511f13fb7e P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:20.279165 6811 raft_consensus.cc:374] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:20.279215 6812 raft_consensus.cc:374] T 08d82302438544e4925a4f511f13fb7e P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:20.279247 6811 raft_consensus.cc:729] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 333acff728274715a9816c286921ddd0, State: Initialized, Role: FOLLOWER
I20240502 14:23:20.279292 6812 raft_consensus.cc:729] T 08d82302438544e4925a4f511f13fb7e P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 181b466de0084fcbace0ca327fb1cbef, State: Initialized, Role: FOLLOWER
I20240502 14:23:20.279474 6811 consensus_queue.cc:260] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:20.280032 6811 ts_tablet_manager.cc:1398] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:23:20.280489 6812 consensus_queue.cc:260] T 08d82302438544e4925a4f511f13fb7e P 181b466de0084fcbace0ca327fb1cbef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:20.281250 6812 ts_tablet_manager.cc:1398] T 08d82302438544e4925a4f511f13fb7e P 181b466de0084fcbace0ca327fb1cbef: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:23:20.286005 6812 tablet_bootstrap.cc:492] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef: Bootstrap starting.
I20240502 14:23:20.287412 6104 tablet_service.cc:1467] Processing CreateTablet for tablet 81cfc8c8608b4cb0b7c303a910791d4b (DEFAULT_TABLE table=rebalance_test_table_1 [id=a3f5812bc1aa47c1990b9a769141e329]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:20.288066 6812 tablet_bootstrap.cc:654] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:20.288432 6104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81cfc8c8608b4cb0b7c303a910791d4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:20.293736 6819 tablet_bootstrap.cc:492] T 3128a2a3296e4940bfdf669fecfd7877 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap starting.
I20240502 14:23:20.294811 6810 tablet_bootstrap.cc:492] T 3128a2a3296e4940bfdf669fecfd7877 P 83e62d4e5f8a4316bc1bcb8c18458081: No bootstrap required, opened a new log
I20240502 14:23:20.294817 6811 tablet_bootstrap.cc:492] T 81cfc8c8608b4cb0b7c303a910791d4b P 333acff728274715a9816c286921ddd0: Bootstrap starting.
I20240502 14:23:20.294961 6810 ts_tablet_manager.cc:1367] T 3128a2a3296e4940bfdf669fecfd7877 P 83e62d4e5f8a4316bc1bcb8c18458081: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.006s
I20240502 14:23:20.295578 6810 raft_consensus.cc:348] T 3128a2a3296e4940bfdf669fecfd7877 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:23:20.295778 6812 tablet_bootstrap.cc:492] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef: No bootstrap required, opened a new log
I20240502 14:23:20.295787 6810 raft_consensus.cc:374] T 3128a2a3296e4940bfdf669fecfd7877 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:20.295858 6810 raft_consensus.cc:729] T 3128a2a3296e4940bfdf669fecfd7877 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e62d4e5f8a4316bc1bcb8c18458081, State: Initialized, Role: FOLLOWER
I20240502 14:23:20.295929 6812 ts_tablet_manager.cc:1367] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20240502 14:23:20.296094 6810 consensus_queue.cc:260] T 3128a2a3296e4940bfdf669fecfd7877 P 83e62d4e5f8a4316bc1bcb8c18458081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:23:20.296254 6811 tablet_bootstrap.cc:654] T 81cfc8c8608b4cb0b7c303a910791d4b P 333acff728274715a9816c286921ddd0: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:20.296577 6810 ts_tablet_manager.cc:1398] T 3128a2a3296e4940bfdf669fecfd7877 P 83e62d4e5f8a4316bc1bcb8c18458081: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:23:20.296823 6812 raft_consensus.cc:348] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:23:20.297026 6810 tablet_bootstrap.cc:492] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081: Bootstrap starting.
I20240502 14:23:20.298004 6810 tablet_bootstrap.cc:654] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:20.297035 6812 raft_consensus.cc:374] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:20.298213 6812 raft_consensus.cc:729] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 181b466de0084fcbace0ca327fb1cbef, State: Initialized, Role: FOLLOWER
I20240502 14:23:20.298527 6812 consensus_queue.cc:260] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:23:20.299121 6812 ts_tablet_manager.cc:1398] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:23:20.299423 6812 tablet_bootstrap.cc:492] T 81cfc8c8608b4cb0b7c303a910791d4b P 181b466de0084fcbace0ca327fb1cbef: Bootstrap starting.
I20240502 14:23:20.301126 6812 tablet_bootstrap.cc:654] T 81cfc8c8608b4cb0b7c303a910791d4b P 181b466de0084fcbace0ca327fb1cbef: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:20.303601 6819 tablet_bootstrap.cc:654] T 3128a2a3296e4940bfdf669fecfd7877 P 1dbf482780eb47e8933983e4f0fd0444: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:20.306078 6812 tablet_bootstrap.cc:492] T 81cfc8c8608b4cb0b7c303a910791d4b P 181b466de0084fcbace0ca327fb1cbef: No bootstrap required, opened a new log
I20240502 14:23:20.306236 6812 ts_tablet_manager.cc:1367] T 81cfc8c8608b4cb0b7c303a910791d4b P 181b466de0084fcbace0ca327fb1cbef: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 14:23:20.307044 6812 raft_consensus.cc:348] T 81cfc8c8608b4cb0b7c303a910791d4b P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } }
I20240502 14:23:20.307281 6812 raft_consensus.cc:374] T 81cfc8c8608b4cb0b7c303a910791d4b P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:20.307377 6812 raft_consensus.cc:729] T 81cfc8c8608b4cb0b7c303a910791d4b P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 181b466de0084fcbace0ca327fb1cbef, State: Initialized, Role: FOLLOWER
I20240502 14:23:20.307641 6812 consensus_queue.cc:260] T 81cfc8c8608b4cb0b7c303a910791d4b P 181b466de0084fcbace0ca327fb1cbef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } }
I20240502 14:23:20.308219 6812 ts_tablet_manager.cc:1398] T 81cfc8c8608b4cb0b7c303a910791d4b P 181b466de0084fcbace0ca327fb1cbef: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 14:23:20.309382 6819 tablet_bootstrap.cc:492] T 3128a2a3296e4940bfdf669fecfd7877 P 1dbf482780eb47e8933983e4f0fd0444: No bootstrap required, opened a new log
I20240502 14:23:20.309592 6819 ts_tablet_manager.cc:1367] T 3128a2a3296e4940bfdf669fecfd7877 P 1dbf482780eb47e8933983e4f0fd0444: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.003s
I20240502 14:23:20.310536 6819 raft_consensus.cc:348] T 3128a2a3296e4940bfdf669fecfd7877 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:23:20.310827 6819 raft_consensus.cc:374] T 3128a2a3296e4940bfdf669fecfd7877 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:20.310964 6819 raft_consensus.cc:729] T 3128a2a3296e4940bfdf669fecfd7877 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbf482780eb47e8933983e4f0fd0444, State: Initialized, Role: FOLLOWER
I20240502 14:23:20.311278 6819 consensus_queue.cc:260] T 3128a2a3296e4940bfdf669fecfd7877 P 1dbf482780eb47e8933983e4f0fd0444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:23:20.312247 6810 tablet_bootstrap.cc:492] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081: No bootstrap required, opened a new log
I20240502 14:23:20.312397 6810 ts_tablet_manager.cc:1367] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20240502 14:23:20.312561 6819 ts_tablet_manager.cc:1398] T 3128a2a3296e4940bfdf669fecfd7877 P 1dbf482780eb47e8933983e4f0fd0444: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:23:20.313041 6810 raft_consensus.cc:348] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } }
I20240502 14:23:20.313171 6810 raft_consensus.cc:374] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:20.313277 6810 raft_consensus.cc:729] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e62d4e5f8a4316bc1bcb8c18458081, State: Initialized, Role: FOLLOWER
I20240502 14:23:20.313455 6810 consensus_queue.cc:260] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } }
I20240502 14:23:20.313880 6810 ts_tablet_manager.cc:1398] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:23:20.314808 6811 tablet_bootstrap.cc:492] T 81cfc8c8608b4cb0b7c303a910791d4b P 333acff728274715a9816c286921ddd0: No bootstrap required, opened a new log
I20240502 14:23:20.314996 6811 ts_tablet_manager.cc:1367] T 81cfc8c8608b4cb0b7c303a910791d4b P 333acff728274715a9816c286921ddd0: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.009s
I20240502 14:23:20.315912 6811 raft_consensus.cc:348] T 81cfc8c8608b4cb0b7c303a910791d4b P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } }
I20240502 14:23:20.316159 6811 raft_consensus.cc:374] T 81cfc8c8608b4cb0b7c303a910791d4b P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:20.316262 6811 raft_consensus.cc:729] T 81cfc8c8608b4cb0b7c303a910791d4b P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 333acff728274715a9816c286921ddd0, State: Initialized, Role: FOLLOWER
I20240502 14:23:20.316493 6811 consensus_queue.cc:260] T 81cfc8c8608b4cb0b7c303a910791d4b P 333acff728274715a9816c286921ddd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } }
I20240502 14:23:20.317131 6811 ts_tablet_manager.cc:1398] T 81cfc8c8608b4cb0b7c303a910791d4b P 333acff728274715a9816c286921ddd0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:23:20.321732 6817 raft_consensus.cc:482] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:23:20.321858 6817 raft_consensus.cc:504] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:23:20.323164 6817 leader_election.cc:288] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015), 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831)
I20240502 14:23:20.343340 6739 raft_consensus.cc:482] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:23:20.343525 6739 raft_consensus.cc:504] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } }
I20240502 14:23:20.343616 5864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3128a2a3296e4940bfdf669fecfd7877" candidate_uuid: "181b466de0084fcbace0ca327fb1cbef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" is_pre_election: true
I20240502 14:23:20.343660 5734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3128a2a3296e4940bfdf669fecfd7877" candidate_uuid: "181b466de0084fcbace0ca327fb1cbef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dbf482780eb47e8933983e4f0fd0444" is_pre_election: true
I20240502 14:23:20.343832 5864 raft_consensus.cc:2454] T 3128a2a3296e4940bfdf669fecfd7877 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 181b466de0084fcbace0ca327fb1cbef in term 0.
I20240502 14:23:20.343850 5734 raft_consensus.cc:2454] T 3128a2a3296e4940bfdf669fecfd7877 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 181b466de0084fcbace0ca327fb1cbef in term 0.
I20240502 14:23:20.344269 6739 leader_election.cc:288] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 181b466de0084fcbace0ca327fb1cbef (127.5.107.67:41517), 333acff728274715a9816c286921ddd0 (127.5.107.68:45913)
I20240502 14:23:20.344327 5927 leader_election.cc:302] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 181b466de0084fcbace0ca327fb1cbef, 1dbf482780eb47e8933983e4f0fd0444; no voters:
I20240502 14:23:20.344604 5993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81cfc8c8608b4cb0b7c303a910791d4b" candidate_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "181b466de0084fcbace0ca327fb1cbef" is_pre_election: true
I20240502 14:23:20.344751 6817 raft_consensus.cc:2781] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:23:20.344785 5993 raft_consensus.cc:2454] T 81cfc8c8608b4cb0b7c303a910791d4b P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e62d4e5f8a4316bc1bcb8c18458081 in term 0.
I20240502 14:23:20.344899 6817 raft_consensus.cc:482] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:23:20.345005 6817 raft_consensus.cc:3037] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:20.345800 5799 leader_election.cc:302] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 181b466de0084fcbace0ca327fb1cbef, 83e62d4e5f8a4316bc1bcb8c18458081; no voters:
I20240502 14:23:20.346069 6739 raft_consensus.cc:2781] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:23:20.346194 6739 raft_consensus.cc:482] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:23:20.346278 6739 raft_consensus.cc:3037] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:20.347925 6817 raft_consensus.cc:504] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:23:20.348469 6817 leader_election.cc:288] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef [CANDIDATE]: Term 1 election: Requested vote from peers 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015), 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831)
I20240502 14:23:20.348918 6739 raft_consensus.cc:504] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } }
I20240502 14:23:20.349083 5734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3128a2a3296e4940bfdf669fecfd7877" candidate_uuid: "181b466de0084fcbace0ca327fb1cbef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dbf482780eb47e8933983e4f0fd0444"
I20240502 14:23:20.349287 5734 raft_consensus.cc:3037] T 3128a2a3296e4940bfdf669fecfd7877 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:20.349529 6739 leader_election.cc:288] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [CANDIDATE]: Term 1 election: Requested vote from peers 181b466de0084fcbace0ca327fb1cbef (127.5.107.67:41517), 333acff728274715a9816c286921ddd0 (127.5.107.68:45913)
I20240502 14:23:20.350036 5993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81cfc8c8608b4cb0b7c303a910791d4b" candidate_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "181b466de0084fcbace0ca327fb1cbef"
I20240502 14:23:20.350061 5864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3128a2a3296e4940bfdf669fecfd7877" candidate_uuid: "181b466de0084fcbace0ca327fb1cbef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081"
I20240502 14:23:20.350194 5993 raft_consensus.cc:3037] T 81cfc8c8608b4cb0b7c303a910791d4b P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:20.350235 5864 raft_consensus.cc:3037] T 3128a2a3296e4940bfdf669fecfd7877 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:20.351698 5734 raft_consensus.cc:2454] T 3128a2a3296e4940bfdf669fecfd7877 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 181b466de0084fcbace0ca327fb1cbef in term 1.
I20240502 14:23:20.352201 5927 leader_election.cc:302] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 181b466de0084fcbace0ca327fb1cbef, 1dbf482780eb47e8933983e4f0fd0444; no voters:
I20240502 14:23:20.352360 5864 raft_consensus.cc:2454] T 3128a2a3296e4940bfdf669fecfd7877 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 181b466de0084fcbace0ca327fb1cbef in term 1.
I20240502 14:23:20.352474 6817 raft_consensus.cc:2781] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:23:20.352614 5993 raft_consensus.cc:2454] T 81cfc8c8608b4cb0b7c303a910791d4b P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e62d4e5f8a4316bc1bcb8c18458081 in term 1.
I20240502 14:23:20.353099 5799 leader_election.cc:302] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 181b466de0084fcbace0ca327fb1cbef, 83e62d4e5f8a4316bc1bcb8c18458081; no voters:
I20240502 14:23:20.353318 6817 raft_consensus.cc:686] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef [term 1 LEADER]: Becoming Leader. State: Replica: 181b466de0084fcbace0ca327fb1cbef, State: Running, Role: LEADER
I20240502 14:23:20.353399 6739 raft_consensus.cc:2781] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:23:20.353821 6123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81cfc8c8608b4cb0b7c303a910791d4b" candidate_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "333acff728274715a9816c286921ddd0" is_pre_election: true
I20240502 14:23:20.353812 6817 consensus_queue.cc:237] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:23:20.353900 6124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81cfc8c8608b4cb0b7c303a910791d4b" candidate_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "333acff728274715a9816c286921ddd0"
I20240502 14:23:20.354040 6123 raft_consensus.cc:2454] T 81cfc8c8608b4cb0b7c303a910791d4b P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e62d4e5f8a4316bc1bcb8c18458081 in term 0.
I20240502 14:23:20.357667 5589 catalog_manager.cc:5484] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef reported cstate change: term changed from 0 to 1, leader changed from <none> to 181b466de0084fcbace0ca327fb1cbef (127.5.107.67). New cstate: current_term: 1 leader_uuid: "181b466de0084fcbace0ca327fb1cbef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20240502 14:23:20.362486 6739 raft_consensus.cc:686] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 LEADER]: Becoming Leader. State: Replica: 83e62d4e5f8a4316bc1bcb8c18458081, State: Running, Role: LEADER
I20240502 14:23:20.362733 6739 consensus_queue.cc:237] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } }
I20240502 14:23:20.365144 5589 catalog_manager.cc:5484] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66). New cstate: current_term: 1 leader_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: UNKNOWN } } }
I20240502 14:23:20.422060 6778 raft_consensus.cc:482] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:23:20.422220 6778 raft_consensus.cc:504] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:20.422895 6778 leader_election.cc:288] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 181b466de0084fcbace0ca327fb1cbef (127.5.107.67:41517), 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015)
I20240502 14:23:20.423460 5864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d82302438544e4925a4f511f13fb7e" candidate_uuid: "333acff728274715a9816c286921ddd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" is_pre_election: true
I20240502 14:23:20.423722 5864 raft_consensus.cc:2454] T 08d82302438544e4925a4f511f13fb7e P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 333acff728274715a9816c286921ddd0 in term 0.
I20240502 14:23:20.424068 6058 leader_election.cc:302] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 333acff728274715a9816c286921ddd0, 83e62d4e5f8a4316bc1bcb8c18458081; no voters:
I20240502 14:23:20.424358 6778 raft_consensus.cc:2781] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:23:20.424470 6778 raft_consensus.cc:482] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:23:20.424557 6778 raft_consensus.cc:3037] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:20.426440 6778 raft_consensus.cc:504] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:20.426963 6778 leader_election.cc:288] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 1 election: Requested vote from peers 181b466de0084fcbace0ca327fb1cbef (127.5.107.67:41517), 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015)
I20240502 14:23:20.427389 5864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d82302438544e4925a4f511f13fb7e" candidate_uuid: "333acff728274715a9816c286921ddd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081"
I20240502 14:23:20.427575 5864 raft_consensus.cc:3037] T 08d82302438544e4925a4f511f13fb7e P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:20.428954 5993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d82302438544e4925a4f511f13fb7e" candidate_uuid: "333acff728274715a9816c286921ddd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "181b466de0084fcbace0ca327fb1cbef" is_pre_election: true
I20240502 14:23:20.429055 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08d82302438544e4925a4f511f13fb7e" candidate_uuid: "333acff728274715a9816c286921ddd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "181b466de0084fcbace0ca327fb1cbef"
I20240502 14:23:20.429224 5993 raft_consensus.cc:2454] T 08d82302438544e4925a4f511f13fb7e P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 333acff728274715a9816c286921ddd0 in term 0.
I20240502 14:23:20.429504 5864 raft_consensus.cc:2454] T 08d82302438544e4925a4f511f13fb7e P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 333acff728274715a9816c286921ddd0 in term 1.
I20240502 14:23:20.429838 6058 leader_election.cc:302] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 333acff728274715a9816c286921ddd0, 83e62d4e5f8a4316bc1bcb8c18458081; no voters: 181b466de0084fcbace0ca327fb1cbef
I20240502 14:23:20.430064 6778 raft_consensus.cc:2781] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:23:20.430346 6778 raft_consensus.cc:686] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [term 1 LEADER]: Becoming Leader. State: Replica: 333acff728274715a9816c286921ddd0, State: Running, Role: LEADER
I20240502 14:23:20.430574 6778 consensus_queue.cc:237] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:20.432554 5589 catalog_manager.cc:5484] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 333acff728274715a9816c286921ddd0 (127.5.107.68). New cstate: current_term: 1 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } health_report { overall_health: UNKNOWN } } }
W20240502 14:23:20.728086 6838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:23:20.728328 6838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:23:20.734730 6838 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:23:20.759099 6830 consensus_queue.cc:1035] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240502 14:23:20.766357 6830 consensus_queue.cc:1035] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:23:20.768501 6124 raft_consensus.cc:3037] T 81cfc8c8608b4cb0b7c303a910791d4b P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:20.771842 6739 consensus_queue.cc:1035] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:23:20.811194 6739 consensus_queue.cc:1035] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:23:20.882905 6778 consensus_queue.cc:1035] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:23:20.896384 5994 raft_consensus.cc:3037] T 08d82302438544e4925a4f511f13fb7e P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:20.899397 5994 raft_consensus.cc:1261] T 08d82302438544e4925a4f511f13fb7e P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:23:20.900096 6778 consensus_queue.cc:1035] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:23:20.926877 6858 mvcc.cc:205] Tried to move back new op lower bound from 7023246544465707008 to 7023246542565429248. Current Snapshot: MvccSnapshot[applied={T|T < 7023246544465707008}]
I20240502 14:23:21.010197 6850 mvcc.cc:205] Tried to move back new op lower bound from 7023246544465707008 to 7023246542565429248. Current Snapshot: MvccSnapshot[applied={T|T < 7023246544465707008 or (T in {7023246544465707008})}]
INSERT report
rows total: 2000
time total: 219.221 ms
time per row: 0.10961 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:23:21.943437 5589 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54758:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:23:21.951900 6104 tablet_service.cc:1467] Processing CreateTablet for tablet 25f9ebd715dc4f96901b60796ea846ad (DEFAULT_TABLE table=rebalance_test_table_2 [id=64688b21b92e4f8596609be0f50e11bf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:21.952351 6104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f9ebd715dc4f96901b60796ea846ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:21.952508 5974 tablet_service.cc:1467] Processing CreateTablet for tablet 9fdda89a7743466a8aeceb543124fd79 (DEFAULT_TABLE table=rebalance_test_table_2 [id=64688b21b92e4f8596609be0f50e11bf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:21.952968 5974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fdda89a7743466a8aeceb543124fd79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:21.953227 6103 tablet_service.cc:1467] Processing CreateTablet for tablet 9fdda89a7743466a8aeceb543124fd79 (DEFAULT_TABLE table=rebalance_test_table_2 [id=64688b21b92e4f8596609be0f50e11bf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:21.953573 6103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fdda89a7743466a8aeceb543124fd79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:21.953480 5713 tablet_service.cc:1467] Processing CreateTablet for tablet 9fdda89a7743466a8aeceb543124fd79 (DEFAULT_TABLE table=rebalance_test_table_2 [id=64688b21b92e4f8596609be0f50e11bf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:21.953835 5713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fdda89a7743466a8aeceb543124fd79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:21.957489 5714 tablet_service.cc:1467] Processing CreateTablet for tablet a6647f23da6c4430a9f84e378dd345a4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=64688b21b92e4f8596609be0f50e11bf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:21.957522 6102 tablet_service.cc:1467] Processing CreateTablet for tablet a6647f23da6c4430a9f84e378dd345a4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=64688b21b92e4f8596609be0f50e11bf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:21.957805 5714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6647f23da6c4430a9f84e378dd345a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:21.957849 6102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6647f23da6c4430a9f84e378dd345a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:21.961412 5712 tablet_service.cc:1467] Processing CreateTablet for tablet 25f9ebd715dc4f96901b60796ea846ad (DEFAULT_TABLE table=rebalance_test_table_2 [id=64688b21b92e4f8596609be0f50e11bf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:21.961722 6907 tablet_bootstrap.cc:492] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0: Bootstrap starting.
I20240502 14:23:21.961783 5712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f9ebd715dc4f96901b60796ea846ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:21.961839 5843 tablet_service.cc:1467] Processing CreateTablet for tablet 25f9ebd715dc4f96901b60796ea846ad (DEFAULT_TABLE table=rebalance_test_table_2 [id=64688b21b92e4f8596609be0f50e11bf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:21.962234 5843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f9ebd715dc4f96901b60796ea846ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:21.964596 6907 tablet_bootstrap.cc:654] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:21.968832 5844 tablet_service.cc:1467] Processing CreateTablet for tablet a6647f23da6c4430a9f84e378dd345a4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=64688b21b92e4f8596609be0f50e11bf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:21.969235 5844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6647f23da6c4430a9f84e378dd345a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:21.970083 6909 tablet_bootstrap.cc:492] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap starting.
I20240502 14:23:21.971479 6909 tablet_bootstrap.cc:654] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:21.972251 6910 tablet_bootstrap.cc:492] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081: Bootstrap starting.
I20240502 14:23:21.973913 6910 tablet_bootstrap.cc:654] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:21.977845 6906 tablet_bootstrap.cc:492] T 9fdda89a7743466a8aeceb543124fd79 P 181b466de0084fcbace0ca327fb1cbef: Bootstrap starting.
I20240502 14:23:21.978031 6907 tablet_bootstrap.cc:492] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0: No bootstrap required, opened a new log
I20240502 14:23:21.978677 6907 ts_tablet_manager.cc:1367] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.006s
I20240502 14:23:21.979279 6906 tablet_bootstrap.cc:654] T 9fdda89a7743466a8aeceb543124fd79 P 181b466de0084fcbace0ca327fb1cbef: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:21.979561 6907 raft_consensus.cc:348] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:21.979843 6907 raft_consensus.cc:374] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:21.979969 6907 raft_consensus.cc:729] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 333acff728274715a9816c286921ddd0, State: Initialized, Role: FOLLOWER
I20240502 14:23:21.980276 6907 consensus_queue.cc:260] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:21.980979 6910 tablet_bootstrap.cc:492] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081: No bootstrap required, opened a new log
I20240502 14:23:21.981053 6907 ts_tablet_manager.cc:1398] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:23:21.981154 6910 ts_tablet_manager.cc:1367] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240502 14:23:21.981388 6907 tablet_bootstrap.cc:492] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0: Bootstrap starting.
I20240502 14:23:21.982066 6910 raft_consensus.cc:348] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:21.982293 6910 raft_consensus.cc:374] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:21.982441 6910 raft_consensus.cc:729] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e62d4e5f8a4316bc1bcb8c18458081, State: Initialized, Role: FOLLOWER
I20240502 14:23:21.982674 6910 consensus_queue.cc:260] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:21.983145 6907 tablet_bootstrap.cc:654] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:21.984230 6910 ts_tablet_manager.cc:1398] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:23:21.984531 6910 tablet_bootstrap.cc:492] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081: Bootstrap starting.
I20240502 14:23:21.986058 6910 tablet_bootstrap.cc:654] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:21.986171 6909 tablet_bootstrap.cc:492] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444: No bootstrap required, opened a new log
I20240502 14:23:21.986359 6909 ts_tablet_manager.cc:1367] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20240502 14:23:21.987264 6909 raft_consensus.cc:348] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:21.987558 6909 raft_consensus.cc:374] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:21.987699 6909 raft_consensus.cc:729] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbf482780eb47e8933983e4f0fd0444, State: Initialized, Role: FOLLOWER
I20240502 14:23:21.987988 6909 consensus_queue.cc:260] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:21.989244 6909 ts_tablet_manager.cc:1398] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:23:21.989428 6910 tablet_bootstrap.cc:492] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081: No bootstrap required, opened a new log
I20240502 14:23:21.989558 6907 tablet_bootstrap.cc:492] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0: No bootstrap required, opened a new log
I20240502 14:23:21.989595 6910 ts_tablet_manager.cc:1367] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240502 14:23:21.989754 6907 ts_tablet_manager.cc:1367] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 14:23:21.990058 6909 tablet_bootstrap.cc:492] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap starting.
I20240502 14:23:21.990468 6907 raft_consensus.cc:348] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:21.990474 6910 raft_consensus.cc:348] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:21.990696 6907 raft_consensus.cc:374] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:21.990789 6907 raft_consensus.cc:729] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 333acff728274715a9816c286921ddd0, State: Initialized, Role: FOLLOWER
I20240502 14:23:21.991047 6907 consensus_queue.cc:260] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:21.991533 6906 tablet_bootstrap.cc:492] T 9fdda89a7743466a8aeceb543124fd79 P 181b466de0084fcbace0ca327fb1cbef: No bootstrap required, opened a new log
I20240502 14:23:21.991580 6907 ts_tablet_manager.cc:1398] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:23:21.991673 6906 ts_tablet_manager.cc:1367] T 9fdda89a7743466a8aeceb543124fd79 P 181b466de0084fcbace0ca327fb1cbef: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20240502 14:23:21.991701 6910 raft_consensus.cc:374] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:21.991781 6910 raft_consensus.cc:729] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e62d4e5f8a4316bc1bcb8c18458081, State: Initialized, Role: FOLLOWER
I20240502 14:23:21.991837 6907 tablet_bootstrap.cc:492] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0: Bootstrap starting.
I20240502 14:23:21.992019 6910 consensus_queue.cc:260] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:21.992337 6906 raft_consensus.cc:348] T 9fdda89a7743466a8aeceb543124fd79 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:21.992528 6906 raft_consensus.cc:374] T 9fdda89a7743466a8aeceb543124fd79 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:21.992615 6906 raft_consensus.cc:729] T 9fdda89a7743466a8aeceb543124fd79 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 181b466de0084fcbace0ca327fb1cbef, State: Initialized, Role: FOLLOWER
I20240502 14:23:21.992821 6909 tablet_bootstrap.cc:654] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:21.992842 6906 consensus_queue.cc:260] T 9fdda89a7743466a8aeceb543124fd79 P 181b466de0084fcbace0ca327fb1cbef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:21.993161 6907 tablet_bootstrap.cc:654] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:21.993182 6910 ts_tablet_manager.cc:1398] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:23:21.993343 6906 ts_tablet_manager.cc:1398] T 9fdda89a7743466a8aeceb543124fd79 P 181b466de0084fcbace0ca327fb1cbef: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:23:21.994246 6909 tablet_bootstrap.cc:492] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444: No bootstrap required, opened a new log
I20240502 14:23:21.994338 6909 ts_tablet_manager.cc:1367] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240502 14:23:21.994838 6909 raft_consensus.cc:348] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:21.994961 6909 raft_consensus.cc:374] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:21.994999 6909 raft_consensus.cc:729] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbf482780eb47e8933983e4f0fd0444, State: Initialized, Role: FOLLOWER
I20240502 14:23:21.995165 6909 consensus_queue.cc:260] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:21.995579 6909 ts_tablet_manager.cc:1398] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:23:21.995846 6909 tablet_bootstrap.cc:492] T 9fdda89a7743466a8aeceb543124fd79 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap starting.
I20240502 14:23:21.996858 6909 tablet_bootstrap.cc:654] T 9fdda89a7743466a8aeceb543124fd79 P 1dbf482780eb47e8933983e4f0fd0444: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:22.001240 6907 tablet_bootstrap.cc:492] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0: No bootstrap required, opened a new log
I20240502 14:23:22.001272 6909 tablet_bootstrap.cc:492] T 9fdda89a7743466a8aeceb543124fd79 P 1dbf482780eb47e8933983e4f0fd0444: No bootstrap required, opened a new log
I20240502 14:23:22.001418 6907 ts_tablet_manager.cc:1367] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.008s
I20240502 14:23:22.001523 6909 ts_tablet_manager.cc:1367] T 9fdda89a7743466a8aeceb543124fd79 P 1dbf482780eb47e8933983e4f0fd0444: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20240502 14:23:22.002106 6907 raft_consensus.cc:348] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:22.002152 6909 raft_consensus.cc:348] T 9fdda89a7743466a8aeceb543124fd79 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:22.002308 6909 raft_consensus.cc:374] T 9fdda89a7743466a8aeceb543124fd79 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:22.002305 6907 raft_consensus.cc:374] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:22.002393 6909 raft_consensus.cc:729] T 9fdda89a7743466a8aeceb543124fd79 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbf482780eb47e8933983e4f0fd0444, State: Initialized, Role: FOLLOWER
I20240502 14:23:22.002394 6907 raft_consensus.cc:729] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 333acff728274715a9816c286921ddd0, State: Initialized, Role: FOLLOWER
I20240502 14:23:22.002604 6909 consensus_queue.cc:260] T 9fdda89a7743466a8aeceb543124fd79 P 1dbf482780eb47e8933983e4f0fd0444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:22.002588 6907 consensus_queue.cc:260] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:22.003144 6907 ts_tablet_manager.cc:1398] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:23:22.003759 6909 ts_tablet_manager.cc:1398] T 9fdda89a7743466a8aeceb543124fd79 P 1dbf482780eb47e8933983e4f0fd0444: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:23:22.031203 6726 raft_consensus.cc:482] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:23:22.031345 6726 raft_consensus.cc:504] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:22.032019 6726 leader_election.cc:288] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 333acff728274715a9816c286921ddd0 (127.5.107.68:45913), 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015)
I20240502 14:23:22.032413 6123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f9ebd715dc4f96901b60796ea846ad" candidate_uuid: "1dbf482780eb47e8933983e4f0fd0444" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "333acff728274715a9816c286921ddd0" is_pre_election: true
I20240502 14:23:22.032430 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f9ebd715dc4f96901b60796ea846ad" candidate_uuid: "1dbf482780eb47e8933983e4f0fd0444" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" is_pre_election: true
I20240502 14:23:22.032590 6123 raft_consensus.cc:2454] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dbf482780eb47e8933983e4f0fd0444 in term 0.
I20240502 14:23:22.032608 5863 raft_consensus.cc:2454] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dbf482780eb47e8933983e4f0fd0444 in term 0.
I20240502 14:23:22.032974 5669 leader_election.cc:302] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dbf482780eb47e8933983e4f0fd0444, 333acff728274715a9816c286921ddd0; no voters:
I20240502 14:23:22.033282 6726 raft_consensus.cc:2781] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:23:22.033370 6726 raft_consensus.cc:482] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:23:22.033445 6726 raft_consensus.cc:3037] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:22.035105 6726 raft_consensus.cc:504] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:22.035547 6726 leader_election.cc:288] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444 [CANDIDATE]: Term 1 election: Requested vote from peers 333acff728274715a9816c286921ddd0 (127.5.107.68:45913), 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015)
I20240502 14:23:22.036047 6123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f9ebd715dc4f96901b60796ea846ad" candidate_uuid: "1dbf482780eb47e8933983e4f0fd0444" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "333acff728274715a9816c286921ddd0"
I20240502 14:23:22.036058 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f9ebd715dc4f96901b60796ea846ad" candidate_uuid: "1dbf482780eb47e8933983e4f0fd0444" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081"
I20240502 14:23:22.036237 6123 raft_consensus.cc:3037] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:22.036237 5863 raft_consensus.cc:3037] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:22.038900 5863 raft_consensus.cc:2454] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dbf482780eb47e8933983e4f0fd0444 in term 1.
I20240502 14:23:22.038900 6123 raft_consensus.cc:2454] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dbf482780eb47e8933983e4f0fd0444 in term 1.
I20240502 14:23:22.039279 5668 leader_election.cc:302] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dbf482780eb47e8933983e4f0fd0444, 83e62d4e5f8a4316bc1bcb8c18458081; no voters:
I20240502 14:23:22.039603 6726 raft_consensus.cc:2781] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:23:22.039909 6726 raft_consensus.cc:686] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444 [term 1 LEADER]: Becoming Leader. State: Replica: 1dbf482780eb47e8933983e4f0fd0444, State: Running, Role: LEADER
I20240502 14:23:22.040177 6726 consensus_queue.cc:237] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:22.042376 5590 catalog_manager.cc:5484] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65). New cstate: current_term: 1 leader_uuid: "1dbf482780eb47e8933983e4f0fd0444" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } health_report { overall_health: UNKNOWN } } }
I20240502 14:23:22.043648 6836 raft_consensus.cc:482] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:23:22.043766 6836 raft_consensus.cc:504] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:22.044339 6836 leader_election.cc:288] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831), 181b466de0084fcbace0ca327fb1cbef (127.5.107.67:41517)
I20240502 14:23:22.044806 5734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdda89a7743466a8aeceb543124fd79" candidate_uuid: "333acff728274715a9816c286921ddd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dbf482780eb47e8933983e4f0fd0444" is_pre_election: true
I20240502 14:23:22.044799 5992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdda89a7743466a8aeceb543124fd79" candidate_uuid: "333acff728274715a9816c286921ddd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "181b466de0084fcbace0ca327fb1cbef" is_pre_election: true
I20240502 14:23:22.044998 5992 raft_consensus.cc:2454] T 9fdda89a7743466a8aeceb543124fd79 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 333acff728274715a9816c286921ddd0 in term 0.
I20240502 14:23:22.044994 5734 raft_consensus.cc:2454] T 9fdda89a7743466a8aeceb543124fd79 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 333acff728274715a9816c286921ddd0 in term 0.
I20240502 14:23:22.045370 6057 leader_election.cc:302] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dbf482780eb47e8933983e4f0fd0444, 333acff728274715a9816c286921ddd0; no voters:
I20240502 14:23:22.045696 6836 raft_consensus.cc:2781] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:23:22.045832 6836 raft_consensus.cc:482] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:23:22.045919 6836 raft_consensus.cc:3037] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:22.047663 6836 raft_consensus.cc:504] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:22.048187 6836 leader_election.cc:288] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 1 election: Requested vote from peers 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831), 181b466de0084fcbace0ca327fb1cbef (127.5.107.67:41517)
I20240502 14:23:22.048524 5734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdda89a7743466a8aeceb543124fd79" candidate_uuid: "333acff728274715a9816c286921ddd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dbf482780eb47e8933983e4f0fd0444"
I20240502 14:23:22.048633 5992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdda89a7743466a8aeceb543124fd79" candidate_uuid: "333acff728274715a9816c286921ddd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "181b466de0084fcbace0ca327fb1cbef"
I20240502 14:23:22.048691 5734 raft_consensus.cc:3037] T 9fdda89a7743466a8aeceb543124fd79 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:22.048787 5992 raft_consensus.cc:3037] T 9fdda89a7743466a8aeceb543124fd79 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:22.050348 5734 raft_consensus.cc:2454] T 9fdda89a7743466a8aeceb543124fd79 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 333acff728274715a9816c286921ddd0 in term 1.
I20240502 14:23:22.050546 5992 raft_consensus.cc:2454] T 9fdda89a7743466a8aeceb543124fd79 P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 333acff728274715a9816c286921ddd0 in term 1.
I20240502 14:23:22.050652 6057 leader_election.cc:302] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dbf482780eb47e8933983e4f0fd0444, 333acff728274715a9816c286921ddd0; no voters:
I20240502 14:23:22.050926 6836 raft_consensus.cc:2781] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:23:22.051255 6836 raft_consensus.cc:686] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [term 1 LEADER]: Becoming Leader. State: Replica: 333acff728274715a9816c286921ddd0, State: Running, Role: LEADER
I20240502 14:23:22.051515 6836 consensus_queue.cc:237] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:22.053645 5590 catalog_manager.cc:5484] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 333acff728274715a9816c286921ddd0 (127.5.107.68). New cstate: current_term: 1 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } health_report { overall_health: UNKNOWN } } }
I20240502 14:23:22.348012 6726 raft_consensus.cc:482] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:23:22.348253 6726 raft_consensus.cc:504] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:22.348824 6726 leader_election.cc:288] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 333acff728274715a9816c286921ddd0 (127.5.107.68:45913), 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015)
I20240502 14:23:22.349300 6123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6647f23da6c4430a9f84e378dd345a4" candidate_uuid: "1dbf482780eb47e8933983e4f0fd0444" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "333acff728274715a9816c286921ddd0" is_pre_election: true
I20240502 14:23:22.349340 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6647f23da6c4430a9f84e378dd345a4" candidate_uuid: "1dbf482780eb47e8933983e4f0fd0444" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" is_pre_election: true
I20240502 14:23:22.349534 6123 raft_consensus.cc:2454] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dbf482780eb47e8933983e4f0fd0444 in term 0.
I20240502 14:23:22.349607 5863 raft_consensus.cc:2454] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dbf482780eb47e8933983e4f0fd0444 in term 0.
I20240502 14:23:22.349923 5669 leader_election.cc:302] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dbf482780eb47e8933983e4f0fd0444, 333acff728274715a9816c286921ddd0; no voters:
I20240502 14:23:22.350216 6726 raft_consensus.cc:2781] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:23:22.350327 6726 raft_consensus.cc:482] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:23:22.350553 6726 raft_consensus.cc:3037] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:22.352727 6726 raft_consensus.cc:504] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:22.353322 6726 leader_election.cc:288] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [CANDIDATE]: Term 1 election: Requested vote from peers 333acff728274715a9816c286921ddd0 (127.5.107.68:45913), 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015)
I20240502 14:23:22.353672 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6647f23da6c4430a9f84e378dd345a4" candidate_uuid: "1dbf482780eb47e8933983e4f0fd0444" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081"
I20240502 14:23:22.353848 5863 raft_consensus.cc:3037] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:22.353809 6123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6647f23da6c4430a9f84e378dd345a4" candidate_uuid: "1dbf482780eb47e8933983e4f0fd0444" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "333acff728274715a9816c286921ddd0"
I20240502 14:23:22.353991 6123 raft_consensus.cc:3037] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:22.355566 5863 raft_consensus.cc:2454] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dbf482780eb47e8933983e4f0fd0444 in term 1.
I20240502 14:23:22.355648 6123 raft_consensus.cc:2454] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dbf482780eb47e8933983e4f0fd0444 in term 1.
I20240502 14:23:22.355919 5668 leader_election.cc:302] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dbf482780eb47e8933983e4f0fd0444, 83e62d4e5f8a4316bc1bcb8c18458081; no voters:
I20240502 14:23:22.356185 6726 raft_consensus.cc:2781] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:23:22.356341 6726 raft_consensus.cc:686] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 LEADER]: Becoming Leader. State: Replica: 1dbf482780eb47e8933983e4f0fd0444, State: Running, Role: LEADER
I20240502 14:23:22.356586 6726 consensus_queue.cc:237] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:22.359217 5590 catalog_manager.cc:5484] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65). New cstate: current_term: 1 leader_uuid: "1dbf482780eb47e8933983e4f0fd0444" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } health_report { overall_health: UNKNOWN } } }
I20240502 14:23:22.517613 6726 consensus_queue.cc:1035] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444 [LEADER]: Connected to new peer: Peer: permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:23:22.522433 6726 consensus_queue.cc:1035] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:23:22.549176 6836 consensus_queue.cc:1035] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:23:22.554129 6836 consensus_queue.cc:1035] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:23:22.723639 6924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:23:22.723942 6924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:23:22.730667 6924 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 14:23:22.860191 6726 consensus_queue.cc:1035] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [LEADER]: Connected to new peer: Peer: permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:23:22.930095 5863 raft_consensus.cc:1261] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Refusing update from remote peer 1dbf482780eb47e8933983e4f0fd0444: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:23:22.930927 6973 consensus_queue.cc:1035] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:23:23.024283 6976 mvcc.cc:205] Tried to move back new op lower bound from 7023246552666681344 to 7023246550454435840. Current Snapshot: MvccSnapshot[applied={T|T < 7023246552666681344}]
I20240502 14:23:23.037413 6964 mvcc.cc:205] Tried to move back new op lower bound from 7023246553181933568 to 7023246550454435840. Current Snapshot: MvccSnapshot[applied={T|T < 7023246553181933568}]
INSERT report
rows total: 2000
time total: 265.507 ms
time per row: 0.132754 ms
I20240502 14:23:23.127919 5587 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54758:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:23:23.136008 5843 tablet_service.cc:1467] Processing CreateTablet for tablet 0697cc9c0d26419083f71a411b2699ed (DEFAULT_TABLE table=rebalance_test_table_3 [id=5a9100430c684a96b7fd7563ec356139]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:23.136484 5843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0697cc9c0d26419083f71a411b2699ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:23.136415 5974 tablet_service.cc:1467] Processing CreateTablet for tablet 0697cc9c0d26419083f71a411b2699ed (DEFAULT_TABLE table=rebalance_test_table_3 [id=5a9100430c684a96b7fd7563ec356139]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:23.136847 5974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0697cc9c0d26419083f71a411b2699ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:23.136895 5712 tablet_service.cc:1467] Processing CreateTablet for tablet 0697cc9c0d26419083f71a411b2699ed (DEFAULT_TABLE table=rebalance_test_table_3 [id=5a9100430c684a96b7fd7563ec356139]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:23.137336 5712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0697cc9c0d26419083f71a411b2699ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:23.140935 5844 tablet_service.cc:1467] Processing CreateTablet for tablet 5358f204e4324db78adad5af90c17b86 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5a9100430c684a96b7fd7563ec356139]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:23.140938 5713 tablet_service.cc:1467] Processing CreateTablet for tablet f6a8b93193d64330bc53a5c3b363f695 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5a9100430c684a96b7fd7563ec356139]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:23.141311 5713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6a8b93193d64330bc53a5c3b363f695. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:23.141314 5844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5358f204e4324db78adad5af90c17b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:23.141570 5972 tablet_service.cc:1467] Processing CreateTablet for tablet f6a8b93193d64330bc53a5c3b363f695 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5a9100430c684a96b7fd7563ec356139]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:23.141929 5972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6a8b93193d64330bc53a5c3b363f695. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:23.144747 5973 tablet_service.cc:1467] Processing CreateTablet for tablet 5358f204e4324db78adad5af90c17b86 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5a9100430c684a96b7fd7563ec356139]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:23.145190 5973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5358f204e4324db78adad5af90c17b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:23.148674 6992 tablet_bootstrap.cc:492] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081: Bootstrap starting.
I20240502 14:23:23.149624 6103 tablet_service.cc:1467] Processing CreateTablet for tablet f6a8b93193d64330bc53a5c3b363f695 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5a9100430c684a96b7fd7563ec356139]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:23.149645 6104 tablet_service.cc:1467] Processing CreateTablet for tablet 5358f204e4324db78adad5af90c17b86 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5a9100430c684a96b7fd7563ec356139]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:23.150066 6103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6a8b93193d64330bc53a5c3b363f695. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:23.150283 6104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5358f204e4324db78adad5af90c17b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:23.150403 6992 tablet_bootstrap.cc:654] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:23.152700 6992 tablet_bootstrap.cc:492] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081: No bootstrap required, opened a new log
I20240502 14:23:23.152807 6992 ts_tablet_manager.cc:1367] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:23:23.153525 6992 raft_consensus.cc:348] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:23:23.153692 6992 raft_consensus.cc:374] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:23.153759 6992 raft_consensus.cc:729] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e62d4e5f8a4316bc1bcb8c18458081, State: Initialized, Role: FOLLOWER
I20240502 14:23:23.153945 6992 consensus_queue.cc:260] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:23:23.154423 6992 ts_tablet_manager.cc:1398] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:23:23.154723 6992 tablet_bootstrap.cc:492] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081: Bootstrap starting.
I20240502 14:23:23.156013 6992 tablet_bootstrap.cc:654] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:23.160413 6994 tablet_bootstrap.cc:492] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0: Bootstrap starting.
I20240502 14:23:23.163851 6994 tablet_bootstrap.cc:654] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:23.165498 6992 tablet_bootstrap.cc:492] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081: No bootstrap required, opened a new log
I20240502 14:23:23.165741 6992 ts_tablet_manager.cc:1367] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.004s
I20240502 14:23:23.165874 6996 tablet_bootstrap.cc:492] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap starting.
I20240502 14:23:23.166759 6992 raft_consensus.cc:348] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:23.167135 6992 raft_consensus.cc:374] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:23.167287 6992 raft_consensus.cc:729] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e62d4e5f8a4316bc1bcb8c18458081, State: Initialized, Role: FOLLOWER
I20240502 14:23:23.167625 6992 consensus_queue.cc:260] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:23.168150 6996 tablet_bootstrap.cc:654] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:23.171243 6991 tablet_bootstrap.cc:492] T 0697cc9c0d26419083f71a411b2699ed P 181b466de0084fcbace0ca327fb1cbef: Bootstrap starting.
I20240502 14:23:23.171511 6992 ts_tablet_manager.cc:1398] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:23:23.173074 6991 tablet_bootstrap.cc:654] T 0697cc9c0d26419083f71a411b2699ed P 181b466de0084fcbace0ca327fb1cbef: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:23.173316 6994 tablet_bootstrap.cc:492] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0: No bootstrap required, opened a new log
I20240502 14:23:23.173496 6994 ts_tablet_manager.cc:1367] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20240502 14:23:23.173976 6996 tablet_bootstrap.cc:492] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444: No bootstrap required, opened a new log
I20240502 14:23:23.174098 6996 ts_tablet_manager.cc:1367] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20240502 14:23:23.174305 6994 raft_consensus.cc:348] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:23.174566 6994 raft_consensus.cc:374] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:23.174674 6994 raft_consensus.cc:729] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 333acff728274715a9816c286921ddd0, State: Initialized, Role: FOLLOWER
I20240502 14:23:23.174851 6996 raft_consensus.cc:348] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:23:23.175050 6996 raft_consensus.cc:374] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:23.175148 6996 raft_consensus.cc:729] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbf482780eb47e8933983e4f0fd0444, State: Initialized, Role: FOLLOWER
I20240502 14:23:23.175029 6994 consensus_queue.cc:260] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:23.175659 6994 ts_tablet_manager.cc:1398] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:23:23.175354 6996 consensus_queue.cc:260] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:23:23.176514 6996 ts_tablet_manager.cc:1398] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:23:23.176853 6996 tablet_bootstrap.cc:492] T f6a8b93193d64330bc53a5c3b363f695 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap starting.
I20240502 14:23:23.178882 6996 tablet_bootstrap.cc:654] T f6a8b93193d64330bc53a5c3b363f695 P 1dbf482780eb47e8933983e4f0fd0444: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:23.185325 6994 tablet_bootstrap.cc:492] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0: Bootstrap starting.
I20240502 14:23:23.187948 6994 tablet_bootstrap.cc:654] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:23.188232 6996 tablet_bootstrap.cc:492] T f6a8b93193d64330bc53a5c3b363f695 P 1dbf482780eb47e8933983e4f0fd0444: No bootstrap required, opened a new log
I20240502 14:23:23.188362 6996 ts_tablet_manager.cc:1367] T f6a8b93193d64330bc53a5c3b363f695 P 1dbf482780eb47e8933983e4f0fd0444: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20240502 14:23:23.190424 6991 tablet_bootstrap.cc:492] T 0697cc9c0d26419083f71a411b2699ed P 181b466de0084fcbace0ca327fb1cbef: No bootstrap required, opened a new log
I20240502 14:23:23.190610 6991 ts_tablet_manager.cc:1367] T 0697cc9c0d26419083f71a411b2699ed P 181b466de0084fcbace0ca327fb1cbef: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.003s
I20240502 14:23:23.190589 6996 raft_consensus.cc:348] T f6a8b93193d64330bc53a5c3b363f695 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } }
I20240502 14:23:23.190876 6996 raft_consensus.cc:374] T f6a8b93193d64330bc53a5c3b363f695 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:23.190981 6996 raft_consensus.cc:729] T f6a8b93193d64330bc53a5c3b363f695 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbf482780eb47e8933983e4f0fd0444, State: Initialized, Role: FOLLOWER
I20240502 14:23:23.191345 6996 consensus_queue.cc:260] T f6a8b93193d64330bc53a5c3b363f695 P 1dbf482780eb47e8933983e4f0fd0444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } }
I20240502 14:23:23.191604 6991 raft_consensus.cc:348] T 0697cc9c0d26419083f71a411b2699ed P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:23:23.191854 6991 raft_consensus.cc:374] T 0697cc9c0d26419083f71a411b2699ed P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:23.191941 6991 raft_consensus.cc:729] T 0697cc9c0d26419083f71a411b2699ed P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 181b466de0084fcbace0ca327fb1cbef, State: Initialized, Role: FOLLOWER
I20240502 14:23:23.192174 6991 consensus_queue.cc:260] T 0697cc9c0d26419083f71a411b2699ed P 181b466de0084fcbace0ca327fb1cbef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:23:23.192893 6991 ts_tablet_manager.cc:1398] T 0697cc9c0d26419083f71a411b2699ed P 181b466de0084fcbace0ca327fb1cbef: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 14:23:23.193472 6991 tablet_bootstrap.cc:492] T f6a8b93193d64330bc53a5c3b363f695 P 181b466de0084fcbace0ca327fb1cbef: Bootstrap starting.
I20240502 14:23:23.195251 6996 ts_tablet_manager.cc:1398] T f6a8b93193d64330bc53a5c3b363f695 P 1dbf482780eb47e8933983e4f0fd0444: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 14:23:23.195466 6991 tablet_bootstrap.cc:654] T f6a8b93193d64330bc53a5c3b363f695 P 181b466de0084fcbace0ca327fb1cbef: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:23.197286 6994 tablet_bootstrap.cc:492] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0: No bootstrap required, opened a new log
I20240502 14:23:23.197455 6994 ts_tablet_manager.cc:1367] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.001s
I20240502 14:23:23.198297 6994 raft_consensus.cc:348] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } }
I20240502 14:23:23.198460 6994 raft_consensus.cc:374] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:23.198532 6994 raft_consensus.cc:729] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 333acff728274715a9816c286921ddd0, State: Initialized, Role: FOLLOWER
I20240502 14:23:23.198711 6991 tablet_bootstrap.cc:492] T f6a8b93193d64330bc53a5c3b363f695 P 181b466de0084fcbace0ca327fb1cbef: No bootstrap required, opened a new log
I20240502 14:23:23.198724 6994 consensus_queue.cc:260] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } }
I20240502 14:23:23.198840 6991 ts_tablet_manager.cc:1367] T f6a8b93193d64330bc53a5c3b363f695 P 181b466de0084fcbace0ca327fb1cbef: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240502 14:23:23.199160 6994 ts_tablet_manager.cc:1398] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:23:23.199429 6991 raft_consensus.cc:348] T f6a8b93193d64330bc53a5c3b363f695 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } }
I20240502 14:23:23.199597 6991 raft_consensus.cc:374] T f6a8b93193d64330bc53a5c3b363f695 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:23.199712 6991 raft_consensus.cc:729] T f6a8b93193d64330bc53a5c3b363f695 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 181b466de0084fcbace0ca327fb1cbef, State: Initialized, Role: FOLLOWER
I20240502 14:23:23.199952 6991 consensus_queue.cc:260] T f6a8b93193d64330bc53a5c3b363f695 P 181b466de0084fcbace0ca327fb1cbef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } }
I20240502 14:23:23.200491 6991 ts_tablet_manager.cc:1398] T f6a8b93193d64330bc53a5c3b363f695 P 181b466de0084fcbace0ca327fb1cbef: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:23:23.200750 6991 tablet_bootstrap.cc:492] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef: Bootstrap starting.
I20240502 14:23:23.202188 6991 tablet_bootstrap.cc:654] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:23.204027 6991 tablet_bootstrap.cc:492] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef: No bootstrap required, opened a new log
I20240502 14:23:23.204131 6991 ts_tablet_manager.cc:1367] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:23:23.204569 6991 raft_consensus.cc:348] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:23.204699 6991 raft_consensus.cc:374] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:23.204777 6991 raft_consensus.cc:729] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 181b466de0084fcbace0ca327fb1cbef, State: Initialized, Role: FOLLOWER
I20240502 14:23:23.204938 6991 consensus_queue.cc:260] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:23.205353 6991 ts_tablet_manager.cc:1398] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:23:23.232527 6883 raft_consensus.cc:482] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:23:23.232679 6883 raft_consensus.cc:504] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:23.233342 6883 leader_election.cc:288] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015), 333acff728274715a9816c286921ddd0 (127.5.107.68:45913)
I20240502 14:23:23.233672 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5358f204e4324db78adad5af90c17b86" candidate_uuid: "181b466de0084fcbace0ca327fb1cbef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" is_pre_election: true
I20240502 14:23:23.233937 5863 raft_consensus.cc:2454] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 181b466de0084fcbace0ca327fb1cbef in term 0.
I20240502 14:23:23.234364 5928 leader_election.cc:302] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 181b466de0084fcbace0ca327fb1cbef, 83e62d4e5f8a4316bc1bcb8c18458081; no voters:
I20240502 14:23:23.234619 6883 raft_consensus.cc:2781] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:23:23.234719 6883 raft_consensus.cc:482] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:23:23.234815 6883 raft_consensus.cc:3037] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:23.237092 6883 raft_consensus.cc:504] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:23.237708 6883 leader_election.cc:288] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [CANDIDATE]: Term 1 election: Requested vote from peers 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015), 333acff728274715a9816c286921ddd0 (127.5.107.68:45913)
I20240502 14:23:23.237954 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5358f204e4324db78adad5af90c17b86" candidate_uuid: "181b466de0084fcbace0ca327fb1cbef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081"
I20240502 14:23:23.238119 5863 raft_consensus.cc:3037] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:23.239795 6123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5358f204e4324db78adad5af90c17b86" candidate_uuid: "181b466de0084fcbace0ca327fb1cbef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "333acff728274715a9816c286921ddd0" is_pre_election: true
I20240502 14:23:23.239990 6123 raft_consensus.cc:2454] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 181b466de0084fcbace0ca327fb1cbef in term 0.
I20240502 14:23:23.239923 6122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5358f204e4324db78adad5af90c17b86" candidate_uuid: "181b466de0084fcbace0ca327fb1cbef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "333acff728274715a9816c286921ddd0"
I20240502 14:23:23.240178 6122 raft_consensus.cc:3037] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:23.240401 5863 raft_consensus.cc:2454] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 181b466de0084fcbace0ca327fb1cbef in term 1.
I20240502 14:23:23.240706 5928 leader_election.cc:302] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 181b466de0084fcbace0ca327fb1cbef, 83e62d4e5f8a4316bc1bcb8c18458081; no voters:
I20240502 14:23:23.240952 6883 raft_consensus.cc:2781] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:23:23.241338 6883 raft_consensus.cc:686] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [term 1 LEADER]: Becoming Leader. State: Replica: 181b466de0084fcbace0ca327fb1cbef, State: Running, Role: LEADER
I20240502 14:23:23.241595 6883 consensus_queue.cc:237] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:23.242250 6122 raft_consensus.cc:2454] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 181b466de0084fcbace0ca327fb1cbef in term 1.
I20240502 14:23:23.243631 5587 catalog_manager.cc:5484] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef reported cstate change: term changed from 0 to 1, leader changed from <none> to 181b466de0084fcbace0ca327fb1cbef (127.5.107.67). New cstate: current_term: 1 leader_uuid: "181b466de0084fcbace0ca327fb1cbef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } health_report { overall_health: HEALTHY } } }
I20240502 14:23:23.283563 6931 raft_consensus.cc:482] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:23:23.283708 6931 raft_consensus.cc:504] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } }
I20240502 14:23:23.284277 6931 leader_election.cc:288] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831), 181b466de0084fcbace0ca327fb1cbef (127.5.107.67:41517)
I20240502 14:23:23.284713 5992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6a8b93193d64330bc53a5c3b363f695" candidate_uuid: "333acff728274715a9816c286921ddd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "181b466de0084fcbace0ca327fb1cbef" is_pre_election: true
I20240502 14:23:23.284732 5733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6a8b93193d64330bc53a5c3b363f695" candidate_uuid: "333acff728274715a9816c286921ddd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dbf482780eb47e8933983e4f0fd0444" is_pre_election: true
I20240502 14:23:23.284940 5992 raft_consensus.cc:2454] T f6a8b93193d64330bc53a5c3b363f695 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 333acff728274715a9816c286921ddd0 in term 0.
I20240502 14:23:23.284973 5733 raft_consensus.cc:2454] T f6a8b93193d64330bc53a5c3b363f695 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 333acff728274715a9816c286921ddd0 in term 0.
I20240502 14:23:23.285331 6057 leader_election.cc:302] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dbf482780eb47e8933983e4f0fd0444, 333acff728274715a9816c286921ddd0; no voters:
I20240502 14:23:23.285549 6931 raft_consensus.cc:2781] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:23:23.285642 6931 raft_consensus.cc:482] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:23:23.285717 6931 raft_consensus.cc:3037] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:23.287513 6931 raft_consensus.cc:504] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } }
I20240502 14:23:23.287947 6931 leader_election.cc:288] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 1 election: Requested vote from peers 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831), 181b466de0084fcbace0ca327fb1cbef (127.5.107.67:41517)
I20240502 14:23:23.288275 5992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6a8b93193d64330bc53a5c3b363f695" candidate_uuid: "333acff728274715a9816c286921ddd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "181b466de0084fcbace0ca327fb1cbef"
I20240502 14:23:23.288295 5733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6a8b93193d64330bc53a5c3b363f695" candidate_uuid: "333acff728274715a9816c286921ddd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dbf482780eb47e8933983e4f0fd0444"
I20240502 14:23:23.288429 5733 raft_consensus.cc:3037] T f6a8b93193d64330bc53a5c3b363f695 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:23.288424 5992 raft_consensus.cc:3037] T f6a8b93193d64330bc53a5c3b363f695 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:23.289999 5992 raft_consensus.cc:2454] T f6a8b93193d64330bc53a5c3b363f695 P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 333acff728274715a9816c286921ddd0 in term 1.
I20240502 14:23:23.290122 5733 raft_consensus.cc:2454] T f6a8b93193d64330bc53a5c3b363f695 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 333acff728274715a9816c286921ddd0 in term 1.
I20240502 14:23:23.290328 6059 leader_election.cc:302] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 181b466de0084fcbace0ca327fb1cbef, 333acff728274715a9816c286921ddd0; no voters:
I20240502 14:23:23.290555 6931 raft_consensus.cc:2781] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:23:23.290663 6931 raft_consensus.cc:686] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 [term 1 LEADER]: Becoming Leader. State: Replica: 333acff728274715a9816c286921ddd0, State: Running, Role: LEADER
I20240502 14:23:23.290869 6931 consensus_queue.cc:237] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } }
I20240502 14:23:23.293043 5587 catalog_manager.cc:5484] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 333acff728274715a9816c286921ddd0 (127.5.107.68). New cstate: current_term: 1 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: HEALTHY } } }
I20240502 14:23:23.357276 6973 raft_consensus.cc:482] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:23:23.357455 6973 raft_consensus.cc:504] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:23:23.358008 6973 leader_election.cc:288] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015), 181b466de0084fcbace0ca327fb1cbef (127.5.107.67:41517)
I20240502 14:23:23.358448 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0697cc9c0d26419083f71a411b2699ed" candidate_uuid: "1dbf482780eb47e8933983e4f0fd0444" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" is_pre_election: true
I20240502 14:23:23.358516 5992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0697cc9c0d26419083f71a411b2699ed" candidate_uuid: "1dbf482780eb47e8933983e4f0fd0444" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "181b466de0084fcbace0ca327fb1cbef" is_pre_election: true
I20240502 14:23:23.358641 5863 raft_consensus.cc:2454] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dbf482780eb47e8933983e4f0fd0444 in term 0.
I20240502 14:23:23.358764 5992 raft_consensus.cc:2454] T 0697cc9c0d26419083f71a411b2699ed P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dbf482780eb47e8933983e4f0fd0444 in term 0.
I20240502 14:23:23.359037 5668 leader_election.cc:302] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dbf482780eb47e8933983e4f0fd0444, 83e62d4e5f8a4316bc1bcb8c18458081; no voters:
I20240502 14:23:23.359305 6973 raft_consensus.cc:2781] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:23:23.359390 6973 raft_consensus.cc:482] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:23:23.359485 6973 raft_consensus.cc:3037] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:23.361142 6973 raft_consensus.cc:504] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:23:23.361703 6973 leader_election.cc:288] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444 [CANDIDATE]: Term 1 election: Requested vote from peers 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015), 181b466de0084fcbace0ca327fb1cbef (127.5.107.67:41517)
I20240502 14:23:23.362038 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0697cc9c0d26419083f71a411b2699ed" candidate_uuid: "1dbf482780eb47e8933983e4f0fd0444" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081"
I20240502 14:23:23.362105 5992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0697cc9c0d26419083f71a411b2699ed" candidate_uuid: "1dbf482780eb47e8933983e4f0fd0444" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "181b466de0084fcbace0ca327fb1cbef"
I20240502 14:23:23.362212 5863 raft_consensus.cc:3037] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:23.362268 5992 raft_consensus.cc:3037] T 0697cc9c0d26419083f71a411b2699ed P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:23.363914 5992 raft_consensus.cc:2454] T 0697cc9c0d26419083f71a411b2699ed P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dbf482780eb47e8933983e4f0fd0444 in term 1.
I20240502 14:23:23.363978 5863 raft_consensus.cc:2454] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dbf482780eb47e8933983e4f0fd0444 in term 1.
I20240502 14:23:23.364320 5669 leader_election.cc:302] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 181b466de0084fcbace0ca327fb1cbef, 1dbf482780eb47e8933983e4f0fd0444; no voters:
I20240502 14:23:23.364557 6973 raft_consensus.cc:2781] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:23:23.364696 6973 raft_consensus.cc:686] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444 [term 1 LEADER]: Becoming Leader. State: Replica: 1dbf482780eb47e8933983e4f0fd0444, State: Running, Role: LEADER
I20240502 14:23:23.364969 6973 consensus_queue.cc:237] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:23:23.367528 5587 catalog_manager.cc:5484] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65). New cstate: current_term: 1 leader_uuid: "1dbf482780eb47e8933983e4f0fd0444" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } health_report { overall_health: HEALTHY } } }
I20240502 14:23:23.635975 6883 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Connected to new peer: Peer: permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:23:23.640975 7005 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:23:23.692392 7009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:23:23.692677 7009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:23:23.699330 7009 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 14:23:23.825371 6973 consensus_queue.cc:1035] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:23:23.827642 6931 consensus_queue.cc:1035] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:23:23.828107 6931 consensus_queue.cc:1035] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:23:23.843204 6973 consensus_queue.cc:1035] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444 [LEADER]: Connected to new peer: Peer: permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:23:23.919983 6057 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.65:44831, user_credentials={real_user=slave}} blocked reactor thread for 92441.6us
INSERT report
rows total: 2000
time total: 282.197 ms
time per row: 0.141098 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:23:24.957841 5589 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54758:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:23:24.966043 5843 tablet_service.cc:1467] Processing CreateTablet for tablet c3ae6199abe742feabe9832da409cc88 (DEFAULT_TABLE table=rebalance_test_table_4 [id=661d0065e6874126812c41b89c61eb98]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:24.966527 5843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3ae6199abe742feabe9832da409cc88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:24.966868 5712 tablet_service.cc:1467] Processing CreateTablet for tablet c3ae6199abe742feabe9832da409cc88 (DEFAULT_TABLE table=rebalance_test_table_4 [id=661d0065e6874126812c41b89c61eb98]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:24.967305 5712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3ae6199abe742feabe9832da409cc88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:24.967561 5713 tablet_service.cc:1467] Processing CreateTablet for tablet 244218906a51489dab769122731b1ec2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=661d0065e6874126812c41b89c61eb98]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:24.967659 5973 tablet_service.cc:1467] Processing CreateTablet for tablet 2f624fefb7de4083b61b38f7b09d3881 (DEFAULT_TABLE table=rebalance_test_table_4 [id=661d0065e6874126812c41b89c61eb98]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:24.967908 5713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 244218906a51489dab769122731b1ec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:24.968096 5973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f624fefb7de4083b61b38f7b09d3881. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:24.969808 5974 tablet_service.cc:1467] Processing CreateTablet for tablet c3ae6199abe742feabe9832da409cc88 (DEFAULT_TABLE table=rebalance_test_table_4 [id=661d0065e6874126812c41b89c61eb98]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:24.970193 5974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3ae6199abe742feabe9832da409cc88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:24.973556 6104 tablet_service.cc:1467] Processing CreateTablet for tablet 244218906a51489dab769122731b1ec2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=661d0065e6874126812c41b89c61eb98]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:24.973982 6104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 244218906a51489dab769122731b1ec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:24.975111 6103 tablet_service.cc:1467] Processing CreateTablet for tablet 2f624fefb7de4083b61b38f7b09d3881 (DEFAULT_TABLE table=rebalance_test_table_4 [id=661d0065e6874126812c41b89c61eb98]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:24.975708 6103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f624fefb7de4083b61b38f7b09d3881. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:24.976081 5844 tablet_service.cc:1467] Processing CreateTablet for tablet 244218906a51489dab769122731b1ec2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=661d0065e6874126812c41b89c61eb98]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:24.976473 5844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 244218906a51489dab769122731b1ec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:24.981319 7079 tablet_bootstrap.cc:492] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap starting.
I20240502 14:23:24.983004 7079 tablet_bootstrap.cc:654] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:24.982431 5713 tablet_service.cc:1467] Processing CreateTablet for tablet 2f624fefb7de4083b61b38f7b09d3881 (DEFAULT_TABLE table=rebalance_test_table_4 [id=661d0065e6874126812c41b89c61eb98]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:23:24.984406 5713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f624fefb7de4083b61b38f7b09d3881. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:24.988201 7080 tablet_bootstrap.cc:492] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef: Bootstrap starting.
I20240502 14:23:24.991199 7080 tablet_bootstrap.cc:654] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:24.993047 7079 tablet_bootstrap.cc:492] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444: No bootstrap required, opened a new log
I20240502 14:23:24.993244 7079 ts_tablet_manager.cc:1367] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.007s
I20240502 14:23:24.993774 7082 tablet_bootstrap.cc:492] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0: Bootstrap starting.
I20240502 14:23:24.994161 7079 raft_consensus.cc:348] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:24.994395 7079 raft_consensus.cc:374] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:24.994530 7079 raft_consensus.cc:729] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbf482780eb47e8933983e4f0fd0444, State: Initialized, Role: FOLLOWER
I20240502 14:23:24.994779 7079 consensus_queue.cc:260] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:24.995427 7082 tablet_bootstrap.cc:654] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:24.995462 7079 ts_tablet_manager.cc:1398] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:23:24.995786 7079 tablet_bootstrap.cc:492] T c3ae6199abe742feabe9832da409cc88 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap starting.
I20240502 14:23:24.996734 7083 tablet_bootstrap.cc:492] T 244218906a51489dab769122731b1ec2 P 83e62d4e5f8a4316bc1bcb8c18458081: Bootstrap starting.
I20240502 14:23:24.997516 7079 tablet_bootstrap.cc:654] T c3ae6199abe742feabe9832da409cc88 P 1dbf482780eb47e8933983e4f0fd0444: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:24.998086 7080 tablet_bootstrap.cc:492] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef: No bootstrap required, opened a new log
I20240502 14:23:24.998250 7080 ts_tablet_manager.cc:1367] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240502 14:23:24.998914 7083 tablet_bootstrap.cc:654] T 244218906a51489dab769122731b1ec2 P 83e62d4e5f8a4316bc1bcb8c18458081: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:25.000268 7080 raft_consensus.cc:348] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:23:25.000530 7080 raft_consensus.cc:374] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:25.000651 7080 raft_consensus.cc:729] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 181b466de0084fcbace0ca327fb1cbef, State: Initialized, Role: FOLLOWER
I20240502 14:23:25.000923 7080 consensus_queue.cc:260] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:23:25.001497 7080 ts_tablet_manager.cc:1398] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:23:25.001531 7083 tablet_bootstrap.cc:492] T 244218906a51489dab769122731b1ec2 P 83e62d4e5f8a4316bc1bcb8c18458081: No bootstrap required, opened a new log
I20240502 14:23:25.001945 7080 tablet_bootstrap.cc:492] T 2f624fefb7de4083b61b38f7b09d3881 P 181b466de0084fcbace0ca327fb1cbef: Bootstrap starting.
I20240502 14:23:25.002502 7082 tablet_bootstrap.cc:492] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0: No bootstrap required, opened a new log
I20240502 14:23:25.002740 7082 ts_tablet_manager.cc:1367] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20240502 14:23:25.003470 7080 tablet_bootstrap.cc:654] T 2f624fefb7de4083b61b38f7b09d3881 P 181b466de0084fcbace0ca327fb1cbef: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:25.003798 7082 raft_consensus.cc:348] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } }
I20240502 14:23:25.004156 7082 raft_consensus.cc:374] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:25.004319 7082 raft_consensus.cc:729] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 333acff728274715a9816c286921ddd0, State: Initialized, Role: FOLLOWER
I20240502 14:23:25.004688 7082 consensus_queue.cc:260] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } }
I20240502 14:23:25.005656 7082 ts_tablet_manager.cc:1398] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:23:25.008244 7082 tablet_bootstrap.cc:492] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0: Bootstrap starting.
I20240502 14:23:25.010366 7082 tablet_bootstrap.cc:654] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:25.010694 7083 ts_tablet_manager.cc:1367] T 244218906a51489dab769122731b1ec2 P 83e62d4e5f8a4316bc1bcb8c18458081: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.001s
I20240502 14:23:25.011339 7080 tablet_bootstrap.cc:492] T 2f624fefb7de4083b61b38f7b09d3881 P 181b466de0084fcbace0ca327fb1cbef: No bootstrap required, opened a new log
I20240502 14:23:25.011551 7080 ts_tablet_manager.cc:1367] T 2f624fefb7de4083b61b38f7b09d3881 P 181b466de0084fcbace0ca327fb1cbef: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240502 14:23:25.011871 7083 raft_consensus.cc:348] T 244218906a51489dab769122731b1ec2 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:25.012164 7083 raft_consensus.cc:374] T 244218906a51489dab769122731b1ec2 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:25.012287 7083 raft_consensus.cc:729] T 244218906a51489dab769122731b1ec2 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e62d4e5f8a4316bc1bcb8c18458081, State: Initialized, Role: FOLLOWER
I20240502 14:23:25.012579 7080 raft_consensus.cc:348] T 2f624fefb7de4083b61b38f7b09d3881 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } }
I20240502 14:23:25.012611 7083 consensus_queue.cc:260] T 244218906a51489dab769122731b1ec2 P 83e62d4e5f8a4316bc1bcb8c18458081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:25.013375 7083 ts_tablet_manager.cc:1398] T 244218906a51489dab769122731b1ec2 P 83e62d4e5f8a4316bc1bcb8c18458081: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:23:25.013777 7083 tablet_bootstrap.cc:492] T c3ae6199abe742feabe9832da409cc88 P 83e62d4e5f8a4316bc1bcb8c18458081: Bootstrap starting.
I20240502 14:23:25.013739 7080 raft_consensus.cc:374] T 2f624fefb7de4083b61b38f7b09d3881 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:25.013921 7080 raft_consensus.cc:729] T 2f624fefb7de4083b61b38f7b09d3881 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 181b466de0084fcbace0ca327fb1cbef, State: Initialized, Role: FOLLOWER
I20240502 14:23:25.014245 7080 consensus_queue.cc:260] T 2f624fefb7de4083b61b38f7b09d3881 P 181b466de0084fcbace0ca327fb1cbef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } }
I20240502 14:23:25.015538 7080 ts_tablet_manager.cc:1398] T 2f624fefb7de4083b61b38f7b09d3881 P 181b466de0084fcbace0ca327fb1cbef: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:23:25.015890 7083 tablet_bootstrap.cc:654] T c3ae6199abe742feabe9832da409cc88 P 83e62d4e5f8a4316bc1bcb8c18458081: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:25.019012 7083 tablet_bootstrap.cc:492] T c3ae6199abe742feabe9832da409cc88 P 83e62d4e5f8a4316bc1bcb8c18458081: No bootstrap required, opened a new log
I20240502 14:23:25.019243 7083 ts_tablet_manager.cc:1367] T c3ae6199abe742feabe9832da409cc88 P 83e62d4e5f8a4316bc1bcb8c18458081: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240502 14:23:25.019519 7079 tablet_bootstrap.cc:492] T c3ae6199abe742feabe9832da409cc88 P 1dbf482780eb47e8933983e4f0fd0444: No bootstrap required, opened a new log
I20240502 14:23:25.019757 7079 ts_tablet_manager.cc:1367] T c3ae6199abe742feabe9832da409cc88 P 1dbf482780eb47e8933983e4f0fd0444: Time spent bootstrapping tablet: real 0.024s user 0.015s sys 0.007s
I20240502 14:23:25.019948 7083 raft_consensus.cc:348] T c3ae6199abe742feabe9832da409cc88 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:23:25.020133 7083 raft_consensus.cc:374] T c3ae6199abe742feabe9832da409cc88 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:25.020208 7083 raft_consensus.cc:729] T c3ae6199abe742feabe9832da409cc88 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e62d4e5f8a4316bc1bcb8c18458081, State: Initialized, Role: FOLLOWER
I20240502 14:23:25.020430 7083 consensus_queue.cc:260] T c3ae6199abe742feabe9832da409cc88 P 83e62d4e5f8a4316bc1bcb8c18458081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:23:25.020979 7079 raft_consensus.cc:348] T c3ae6199abe742feabe9832da409cc88 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:23:25.021318 7079 raft_consensus.cc:374] T c3ae6199abe742feabe9832da409cc88 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:25.021456 7079 raft_consensus.cc:729] T c3ae6199abe742feabe9832da409cc88 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbf482780eb47e8933983e4f0fd0444, State: Initialized, Role: FOLLOWER
I20240502 14:23:25.021790 7083 ts_tablet_manager.cc:1398] T c3ae6199abe742feabe9832da409cc88 P 83e62d4e5f8a4316bc1bcb8c18458081: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:23:25.021628 7082 tablet_bootstrap.cc:492] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0: No bootstrap required, opened a new log
I20240502 14:23:25.021775 7079 consensus_queue.cc:260] T c3ae6199abe742feabe9832da409cc88 P 1dbf482780eb47e8933983e4f0fd0444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:23:25.022045 7082 ts_tablet_manager.cc:1367] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.001s
I20240502 14:23:25.022372 7079 ts_tablet_manager.cc:1398] T c3ae6199abe742feabe9832da409cc88 P 1dbf482780eb47e8933983e4f0fd0444: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:23:25.022682 7079 tablet_bootstrap.cc:492] T 2f624fefb7de4083b61b38f7b09d3881 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap starting.
I20240502 14:23:25.022819 7082 raft_consensus.cc:348] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:25.023029 7082 raft_consensus.cc:374] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:25.023097 7082 raft_consensus.cc:729] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 333acff728274715a9816c286921ddd0, State: Initialized, Role: FOLLOWER
I20240502 14:23:25.023945 7082 consensus_queue.cc:260] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:25.024317 7079 tablet_bootstrap.cc:654] T 2f624fefb7de4083b61b38f7b09d3881 P 1dbf482780eb47e8933983e4f0fd0444: Neither blocks nor log segments found. Creating new log.
I20240502 14:23:25.026216 7079 tablet_bootstrap.cc:492] T 2f624fefb7de4083b61b38f7b09d3881 P 1dbf482780eb47e8933983e4f0fd0444: No bootstrap required, opened a new log
I20240502 14:23:25.026424 7079 ts_tablet_manager.cc:1367] T 2f624fefb7de4083b61b38f7b09d3881 P 1dbf482780eb47e8933983e4f0fd0444: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240502 14:23:25.026629 7082 ts_tablet_manager.cc:1398] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:23:25.027248 7079 raft_consensus.cc:348] T 2f624fefb7de4083b61b38f7b09d3881 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } }
I20240502 14:23:25.027576 7079 raft_consensus.cc:374] T 2f624fefb7de4083b61b38f7b09d3881 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:23:25.027738 7079 raft_consensus.cc:729] T 2f624fefb7de4083b61b38f7b09d3881 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbf482780eb47e8933983e4f0fd0444, State: Initialized, Role: FOLLOWER
I20240502 14:23:25.028015 7079 consensus_queue.cc:260] T 2f624fefb7de4083b61b38f7b09d3881 P 1dbf482780eb47e8933983e4f0fd0444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } }
I20240502 14:23:25.028617 7079 ts_tablet_manager.cc:1398] T 2f624fefb7de4083b61b38f7b09d3881 P 1dbf482780eb47e8933983e4f0fd0444: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:23:25.085661 7005 raft_consensus.cc:482] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:23:25.085829 7005 raft_consensus.cc:504] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:23:25.086406 7005 leader_election.cc:288] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015), 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831)
I20240502 14:23:25.086882 5733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ae6199abe742feabe9832da409cc88" candidate_uuid: "181b466de0084fcbace0ca327fb1cbef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dbf482780eb47e8933983e4f0fd0444" is_pre_election: true
I20240502 14:23:25.086902 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ae6199abe742feabe9832da409cc88" candidate_uuid: "181b466de0084fcbace0ca327fb1cbef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" is_pre_election: true
I20240502 14:23:25.087142 5733 raft_consensus.cc:2454] T c3ae6199abe742feabe9832da409cc88 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 181b466de0084fcbace0ca327fb1cbef in term 0.
I20240502 14:23:25.087142 5863 raft_consensus.cc:2454] T c3ae6199abe742feabe9832da409cc88 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 181b466de0084fcbace0ca327fb1cbef in term 0.
I20240502 14:23:25.087581 5927 leader_election.cc:302] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 181b466de0084fcbace0ca327fb1cbef, 1dbf482780eb47e8933983e4f0fd0444; no voters:
I20240502 14:23:25.087817 7005 raft_consensus.cc:2781] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:23:25.087906 7005 raft_consensus.cc:482] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:23:25.088023 7005 raft_consensus.cc:3037] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:25.089779 7005 raft_consensus.cc:504] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:23:25.090338 7005 leader_election.cc:288] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef [CANDIDATE]: Term 1 election: Requested vote from peers 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015), 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831)
I20240502 14:23:25.090641 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ae6199abe742feabe9832da409cc88" candidate_uuid: "181b466de0084fcbace0ca327fb1cbef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081"
I20240502 14:23:25.090736 5733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ae6199abe742feabe9832da409cc88" candidate_uuid: "181b466de0084fcbace0ca327fb1cbef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dbf482780eb47e8933983e4f0fd0444"
I20240502 14:23:25.090835 5863 raft_consensus.cc:3037] T c3ae6199abe742feabe9832da409cc88 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:25.090937 5733 raft_consensus.cc:3037] T c3ae6199abe742feabe9832da409cc88 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:25.093351 5733 raft_consensus.cc:2454] T c3ae6199abe742feabe9832da409cc88 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 181b466de0084fcbace0ca327fb1cbef in term 1.
I20240502 14:23:25.093351 5863 raft_consensus.cc:2454] T c3ae6199abe742feabe9832da409cc88 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 181b466de0084fcbace0ca327fb1cbef in term 1.
I20240502 14:23:25.093686 5927 leader_election.cc:302] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 181b466de0084fcbace0ca327fb1cbef, 1dbf482780eb47e8933983e4f0fd0444; no voters:
I20240502 14:23:25.093946 7005 raft_consensus.cc:2781] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:23:25.094241 7005 raft_consensus.cc:686] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef [term 1 LEADER]: Becoming Leader. State: Replica: 181b466de0084fcbace0ca327fb1cbef, State: Running, Role: LEADER
I20240502 14:23:25.094455 7005 consensus_queue.cc:237] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:23:25.096774 5589 catalog_manager.cc:5484] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef reported cstate change: term changed from 0 to 1, leader changed from <none> to 181b466de0084fcbace0ca327fb1cbef (127.5.107.67). New cstate: current_term: 1 leader_uuid: "181b466de0084fcbace0ca327fb1cbef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20240502 14:23:25.164986 7064 raft_consensus.cc:482] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:23:25.165169 7064 raft_consensus.cc:504] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:25.165685 7064 leader_election.cc:288] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 333acff728274715a9816c286921ddd0 (127.5.107.68:45913), 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015)
I20240502 14:23:25.166124 6123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "244218906a51489dab769122731b1ec2" candidate_uuid: "1dbf482780eb47e8933983e4f0fd0444" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "333acff728274715a9816c286921ddd0" is_pre_election: true
I20240502 14:23:25.166163 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "244218906a51489dab769122731b1ec2" candidate_uuid: "1dbf482780eb47e8933983e4f0fd0444" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" is_pre_election: true
I20240502 14:23:25.166347 6123 raft_consensus.cc:2454] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dbf482780eb47e8933983e4f0fd0444 in term 0.
I20240502 14:23:25.166363 5863 raft_consensus.cc:2454] T 244218906a51489dab769122731b1ec2 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dbf482780eb47e8933983e4f0fd0444 in term 0.
I20240502 14:23:25.166766 5668 leader_election.cc:302] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1dbf482780eb47e8933983e4f0fd0444, 333acff728274715a9816c286921ddd0, 83e62d4e5f8a4316bc1bcb8c18458081; no voters:
I20240502 14:23:25.167042 7064 raft_consensus.cc:2781] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:23:25.167138 7064 raft_consensus.cc:482] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:23:25.167214 7064 raft_consensus.cc:3037] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:25.168917 7064 raft_consensus.cc:504] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:25.169450 7064 leader_election.cc:288] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444 [CANDIDATE]: Term 1 election: Requested vote from peers 333acff728274715a9816c286921ddd0 (127.5.107.68:45913), 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015)
I20240502 14:23:25.169770 6123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "244218906a51489dab769122731b1ec2" candidate_uuid: "1dbf482780eb47e8933983e4f0fd0444" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "333acff728274715a9816c286921ddd0"
I20240502 14:23:25.169917 6123 raft_consensus.cc:3037] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:25.169858 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "244218906a51489dab769122731b1ec2" candidate_uuid: "1dbf482780eb47e8933983e4f0fd0444" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081"
I20240502 14:23:25.170063 5863 raft_consensus.cc:3037] T 244218906a51489dab769122731b1ec2 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:25.171907 6123 raft_consensus.cc:2454] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dbf482780eb47e8933983e4f0fd0444 in term 1.
I20240502 14:23:25.172153 5863 raft_consensus.cc:2454] T 244218906a51489dab769122731b1ec2 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dbf482780eb47e8933983e4f0fd0444 in term 1.
I20240502 14:23:25.172235 5669 leader_election.cc:302] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dbf482780eb47e8933983e4f0fd0444, 333acff728274715a9816c286921ddd0; no voters:
I20240502 14:23:25.172487 7064 raft_consensus.cc:2781] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:23:25.172824 7064 raft_consensus.cc:686] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 LEADER]: Becoming Leader. State: Replica: 1dbf482780eb47e8933983e4f0fd0444, State: Running, Role: LEADER
I20240502 14:23:25.173060 7064 consensus_queue.cc:237] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:25.175107 5589 catalog_manager.cc:5484] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65). New cstate: current_term: 1 leader_uuid: "1dbf482780eb47e8933983e4f0fd0444" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } health_report { overall_health: UNKNOWN } } }
I20240502 14:23:25.193336 6931 raft_consensus.cc:482] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:23:25.193491 6931 raft_consensus.cc:504] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } }
I20240502 14:23:25.194006 6931 leader_election.cc:288] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831), 181b466de0084fcbace0ca327fb1cbef (127.5.107.67:41517)
I20240502 14:23:25.194375 5992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f624fefb7de4083b61b38f7b09d3881" candidate_uuid: "333acff728274715a9816c286921ddd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "181b466de0084fcbace0ca327fb1cbef" is_pre_election: true
I20240502 14:23:25.194363 5733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f624fefb7de4083b61b38f7b09d3881" candidate_uuid: "333acff728274715a9816c286921ddd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dbf482780eb47e8933983e4f0fd0444" is_pre_election: true
I20240502 14:23:25.194537 5733 raft_consensus.cc:2454] T 2f624fefb7de4083b61b38f7b09d3881 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 333acff728274715a9816c286921ddd0 in term 0.
I20240502 14:23:25.194550 5992 raft_consensus.cc:2454] T 2f624fefb7de4083b61b38f7b09d3881 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 333acff728274715a9816c286921ddd0 in term 0.
I20240502 14:23:25.194857 6057 leader_election.cc:302] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dbf482780eb47e8933983e4f0fd0444, 333acff728274715a9816c286921ddd0; no voters:
I20240502 14:23:25.195082 6931 raft_consensus.cc:2781] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:23:25.195154 6931 raft_consensus.cc:482] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:23:25.195235 6931 raft_consensus.cc:3037] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:25.196870 6931 raft_consensus.cc:504] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } }
I20240502 14:23:25.197439 6931 leader_election.cc:288] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 1 election: Requested vote from peers 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831), 181b466de0084fcbace0ca327fb1cbef (127.5.107.67:41517)
I20240502 14:23:25.197752 5733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f624fefb7de4083b61b38f7b09d3881" candidate_uuid: "333acff728274715a9816c286921ddd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dbf482780eb47e8933983e4f0fd0444"
I20240502 14:23:25.197754 5992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f624fefb7de4083b61b38f7b09d3881" candidate_uuid: "333acff728274715a9816c286921ddd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "181b466de0084fcbace0ca327fb1cbef"
I20240502 14:23:25.197887 5992 raft_consensus.cc:3037] T 2f624fefb7de4083b61b38f7b09d3881 P 181b466de0084fcbace0ca327fb1cbef [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:25.197903 5733 raft_consensus.cc:3037] T 2f624fefb7de4083b61b38f7b09d3881 P 1dbf482780eb47e8933983e4f0fd0444 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:23:25.199410 5733 raft_consensus.cc:2454] T 2f624fefb7de4083b61b38f7b09d3881 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 333acff728274715a9816c286921ddd0 in term 1.
I20240502 14:23:25.199731 5992 raft_consensus.cc:2454] T 2f624fefb7de4083b61b38f7b09d3881 P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 333acff728274715a9816c286921ddd0 in term 1.
I20240502 14:23:25.199730 6057 leader_election.cc:302] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dbf482780eb47e8933983e4f0fd0444, 333acff728274715a9816c286921ddd0; no voters:
I20240502 14:23:25.199993 6931 raft_consensus.cc:2781] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:23:25.200335 6931 raft_consensus.cc:686] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 [term 1 LEADER]: Becoming Leader. State: Replica: 333acff728274715a9816c286921ddd0, State: Running, Role: LEADER
I20240502 14:23:25.200589 6931 consensus_queue.cc:237] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } }
I20240502 14:23:25.203007 5589 catalog_manager.cc:5484] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 333acff728274715a9816c286921ddd0 (127.5.107.68). New cstate: current_term: 1 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: HEALTHY } } }
W20240502 14:23:25.505514 7098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:23:25.505887 7098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:23:25.513391 7098 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:23:25.539431 7005 consensus_queue.cc:1035] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240502 14:23:25.558992 7005 consensus_queue.cc:1035] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:23:25.564920 7064 consensus_queue.cc:1035] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444 [LEADER]: Connected to new peer: Peer: permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:23:25.583204 7094 consensus_queue.cc:1035] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:23:25.583376 7096 consensus_queue.cc:1035] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:23:25.600332 6931 consensus_queue.cc:1035] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 252.859 ms
time per row: 0.12643 ms
I20240502 14:23:25.952350 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.69:37277
--local_ip_for_outbound_sockets=127.5.107.69
--tserver_master_addrs=127.5.107.126:34027
--webserver_port=34655
--webserver_interface=127.5.107.69
--builtin_ntp_servers=127.5.107.84:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 14:23:26.256279 7160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:23:26.256528 7160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:23:26.256620 7160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:23:26.263284 7160 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:23:26.263372 7160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:23:26.263473 7160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.69
I20240502 14:23:26.271400 7160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.69:37277
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.5.107.69
--webserver_port=34655
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.5.107.126:34027
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.69
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:23:26.272526 7160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:23:26.274159 7160 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:23:26.285001 7165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:23:26.285522 7166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:23:26.286155 7168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:23:26.286813 7160 server_base.cc:1034] running on GCE node
I20240502 14:23:26.287333 7160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:23:26.288139 7160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:23:26.289289 7160 hybrid_clock.cc:648] HybridClock initialized: now 1714659806289243 us; error 33 us; skew 500 ppm
I20240502 14:23:26.289582 7160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:23:26.291469 7160 webserver.cc:460] Webserver started at http://127.5.107.69:34655/ using document root <none> and password file <none>
I20240502 14:23:26.292094 7160 fs_manager.cc:359] Metadata directory not provided
I20240502 14:23:26.292207 7160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:23:26.296638 7160 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20240502 14:23:26.299099 7174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:26.300127 7160 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:23:26.300252 7160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "3416b273466c495298342b6e892015c2"
format_stamp: "Formatted at 2024-05-02 14:23:12 on dist-test-slave-2cw1"
I20240502 14:23:26.300961 7160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:23:26.315485 7160 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:23:26.316124 7160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:23:26.316401 7160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:23:26.317327 7160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:23:26.318854 7160 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:23:26.318918 7160 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:26.319020 7160 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:23:26.319064 7160 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:26.349331 7160 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.69:37277
I20240502 14:23:26.349390 7286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.69:37277 every 8 connection(s)
I20240502 14:23:26.350823 7160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:23:26.357539 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 7160
I20240502 14:23:26.358109 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.70:40027
--local_ip_for_outbound_sockets=127.5.107.70
--tserver_master_addrs=127.5.107.126:34027
--webserver_port=39573
--webserver_interface=127.5.107.70
--builtin_ntp_servers=127.5.107.84:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 14:23:26.363093 7287 heartbeater.cc:347] Connected to a master server at 127.5.107.126:34027
I20240502 14:23:26.363348 7287 heartbeater.cc:464] Registering TS with master...
I20240502 14:23:26.364066 7287 heartbeater.cc:511] Master 127.5.107.126:34027 requested a full tablet report, sending...
I20240502 14:23:26.365415 5589 ts_manager.cc:194] Re-registered known tserver with Master: 3416b273466c495298342b6e892015c2 (127.5.107.69:37277)
I20240502 14:23:26.366181 5589 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.69:58265
W20240502 14:23:26.653393 7291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:23:26.653640 7291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:23:26.653779 7291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:23:26.660571 7291 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:23:26.660692 7291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:23:26.660859 7291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.70
I20240502 14:23:26.669289 7291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.70:40027
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.5.107.70
--webserver_port=39573
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.5.107.126:34027
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.70
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:23:26.670419 7291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:23:26.671914 7291 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:23:26.682651 7297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:23:26.683722 7298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:23:26.684160 7300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:23:26.684912 7291 server_base.cc:1034] running on GCE node
I20240502 14:23:26.685395 7291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:23:26.686100 7291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:23:26.687273 7291 hybrid_clock.cc:648] HybridClock initialized: now 1714659806687233 us; error 49 us; skew 500 ppm
I20240502 14:23:26.687541 7291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:23:26.689471 7291 webserver.cc:460] Webserver started at http://127.5.107.70:39573/ using document root <none> and password file <none>
I20240502 14:23:26.690083 7291 fs_manager.cc:359] Metadata directory not provided
I20240502 14:23:26.690174 7291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:23:26.694794 7291 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:23:26.697805 7306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:26.699216 7291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:23:26.699381 7291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "806206bb049040bf8db48a7d8d3840c1"
format_stamp: "Formatted at 2024-05-02 14:23:13 on dist-test-slave-2cw1"
I20240502 14:23:26.700225 7291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:23:26.714111 7291 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:23:26.714687 7291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:23:26.714897 7291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:23:26.715587 7291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:23:26.716938 7291 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:23:26.717007 7291 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:26.717099 7291 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:23:26.717142 7291 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:26.743738 7291 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.70:40027
I20240502 14:23:26.743784 7418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.70:40027 every 8 connection(s)
I20240502 14:23:26.744982 7291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 14:23:26.753981 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 7291
I20240502 14:23:26.754648 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.71:42329
--local_ip_for_outbound_sockets=127.5.107.71
--tserver_master_addrs=127.5.107.126:34027
--webserver_port=35123
--webserver_interface=127.5.107.71
--builtin_ntp_servers=127.5.107.84:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 14:23:26.758656 7419 heartbeater.cc:347] Connected to a master server at 127.5.107.126:34027
I20240502 14:23:26.758922 7419 heartbeater.cc:464] Registering TS with master...
I20240502 14:23:26.759846 7419 heartbeater.cc:511] Master 127.5.107.126:34027 requested a full tablet report, sending...
I20240502 14:23:26.761330 5589 ts_manager.cc:194] Re-registered known tserver with Master: 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70:40027)
I20240502 14:23:26.762418 5589 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.70:34881
W20240502 14:23:27.056198 7422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:23:27.056468 7422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:23:27.056596 7422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:23:27.063970 7422 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:23:27.064086 7422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:23:27.064227 7422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.71
I20240502 14:23:27.072093 7422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.71:42329
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.5.107.71
--webserver_port=35123
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.5.107.126:34027
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.71
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:23:27.073326 7422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:23:27.075002 7422 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:23:27.090354 7427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:23:27.090869 7428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:23:27.091012 7430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:23:27.091786 7422 server_base.cc:1034] running on GCE node
I20240502 14:23:27.092232 7422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:23:27.093117 7422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:23:27.094324 7422 hybrid_clock.cc:648] HybridClock initialized: now 1714659807094280 us; error 37 us; skew 500 ppm
I20240502 14:23:27.094674 7422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:23:27.097086 7422 webserver.cc:460] Webserver started at http://127.5.107.71:35123/ using document root <none> and password file <none>
I20240502 14:23:27.097795 7422 fs_manager.cc:359] Metadata directory not provided
I20240502 14:23:27.097913 7422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:23:27.103181 7422 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 14:23:27.106412 7436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:27.108040 7422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:23:27.108191 7422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "681fd3c49e914094bf28a7df155cffc9"
format_stamp: "Formatted at 2024-05-02 14:23:13 on dist-test-slave-2cw1"
I20240502 14:23:27.108901 7422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:23:27.125012 7422 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:23:27.125694 7422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:23:27.125936 7422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:23:27.126813 7422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:23:27.128290 7422 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:23:27.128355 7422 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:27.128456 7422 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:23:27.128500 7422 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:27.156674 7422 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.71:42329
I20240502 14:23:27.156728 7548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.71:42329 every 8 connection(s)
I20240502 14:23:27.158008 7422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 14:23:27.162171 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 7422
I20240502 14:23:27.162853 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.72:35323
--local_ip_for_outbound_sockets=127.5.107.72
--tserver_master_addrs=127.5.107.126:34027
--webserver_port=36735
--webserver_interface=127.5.107.72
--builtin_ntp_servers=127.5.107.84:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 14:23:27.170428 7549 heartbeater.cc:347] Connected to a master server at 127.5.107.126:34027
I20240502 14:23:27.170727 7549 heartbeater.cc:464] Registering TS with master...
I20240502 14:23:27.171597 7549 heartbeater.cc:511] Master 127.5.107.126:34027 requested a full tablet report, sending...
I20240502 14:23:27.172968 5589 ts_manager.cc:194] Re-registered known tserver with Master: 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71:42329)
I20240502 14:23:27.173764 5589 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.71:53129
I20240502 14:23:27.368534 7287 heartbeater.cc:503] Master 127.5.107.126:34027 was elected leader, sending a full tablet report...
W20240502 14:23:27.468647 7552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:23:27.468873 7552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:23:27.468955 7552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:23:27.475487 7552 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:23:27.475594 7552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:23:27.475718 7552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.72
I20240502 14:23:27.485404 7552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.72:35323
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.5.107.72
--webserver_port=36735
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.5.107.126:34027
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.72
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:23:27.486450 7552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:23:27.487983 7552 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:23:27.497979 7557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:23:27.498597 7560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:23:27.499209 7558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:23:27.499435 7552 server_base.cc:1034] running on GCE node
I20240502 14:23:27.499886 7552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:23:27.500605 7552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:23:27.501772 7552 hybrid_clock.cc:648] HybridClock initialized: now 1714659807501718 us; error 50 us; skew 500 ppm
I20240502 14:23:27.502027 7552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:23:27.503885 7552 webserver.cc:460] Webserver started at http://127.5.107.72:36735/ using document root <none> and password file <none>
I20240502 14:23:27.504487 7552 fs_manager.cc:359] Metadata directory not provided
I20240502 14:23:27.504590 7552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:23:27.509069 7552 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:23:27.511502 7566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:27.512542 7552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:23:27.512670 7552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "83b0e3c0ba724317832f8505a1ffb66e"
format_stamp: "Formatted at 2024-05-02 14:23:14 on dist-test-slave-2cw1"
I20240502 14:23:27.513402 7552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:23:27.533578 7552 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:23:27.534384 7552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:23:27.534699 7552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:23:27.535630 7552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:23:27.537035 7552 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:23:27.537103 7552 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:23:27.537226 7552 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:23:27.537279 7552 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20240502 14:23:27.567545 7552 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.72:35323
I20240502 14:23:27.567607 7678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.72:35323 every 8 connection(s)
I20240502 14:23:27.569080 7552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 14:23:27.570114 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 7552
I20240502 14:23:27.582813 7679 heartbeater.cc:347] Connected to a master server at 127.5.107.126:34027
I20240502 14:23:27.583082 7679 heartbeater.cc:464] Registering TS with master...
I20240502 14:23:27.583786 7679 heartbeater.cc:511] Master 127.5.107.126:34027 requested a full tablet report, sending...
I20240502 14:23:27.585147 5589 ts_manager.cc:194] Re-registered known tserver with Master: 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72:35323)
I20240502 14:23:27.586308 5589 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.72:33025
I20240502 14:23:27.765022 7419 heartbeater.cc:503] Master 127.5.107.126:34027 was elected leader, sending a full tablet report...
I20240502 14:23:28.021401 5973 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:23:28.033988 5713 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:23:28.055137 6103 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:23:28.067955 7614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:28.089725 7484 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:28.103938 5844 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:23:28.112020 7354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:28.118148 7222 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:28.176399 7549 heartbeater.cc:503] Master 127.5.107.126:34027 was elected leader, sending a full tablet report...
I20240502 14:23:28.589236 7679 heartbeater.cc:503] Master 127.5.107.126:34027 was elected leader, sending a full tablet report...
I20240502 14:23:28.615723 5586 ts_manager.cc:292] Set tserver state for 1dbf482780eb47e8933983e4f0fd0444 to MAINTENANCE_MODE
W20240502 14:23:29.922052 7749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:23:29.922322 7749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:23:29.929379 7749 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:23:30.100301 5973 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:23:30.101612 5973 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:23:30.134930 5713 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:23:30.164570 7484 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:30.189850 7222 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:30.189939 7484 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:30.197510 5713 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:23:30.226749 6103 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:23:30.242918 7614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:30.249572 7222 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:30.254036 6103 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:23:30.260454 7354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:30.306540 7614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:30.313716 7354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:30.381430 5844 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:23:30.382081 5844 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:23:30.551729 5973 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:23:30.571281 5713 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:23:30.571616 6103 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:23:30.572316 7222 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:30.581503 7354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:30.597097 7614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:30.597997 7484 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:30.627959 5844 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:23:30.869760 5713 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:23:30.869954 5973 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:23:30.874645 7222 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:30.911440 7484 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:30.918021 6103 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:23:30.964890 7354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:30.965087 7614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:30.985337 5844 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:23:31.010614 7877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:23:31.011113 7877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:23:31.022904 7877 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:23:31.069058 5733 consensus_queue.cc:237] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:31.071182 6123 raft_consensus.cc:1261] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Refusing update from remote peer 1dbf482780eb47e8933983e4f0fd0444: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:23:31.071545 5864 raft_consensus.cc:1261] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Refusing update from remote peer 1dbf482780eb47e8933983e4f0fd0444: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:23:31.071741 7931 consensus_queue.cc:1035] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [LEADER]: Connected to new peer: Peer: permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:23:31.072104 7094 consensus_queue.cc:1035] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:23:31.075217 7931 raft_consensus.cc:2932] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } }
I20240502 14:23:31.076331 5864 raft_consensus.cc:2932] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } }
I20240502 14:23:31.079025 6123 raft_consensus.cc:2932] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } }
I20240502 14:23:31.082064 5591 catalog_manager.cc:5484] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "1dbf482780eb47e8933983e4f0fd0444" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } }
I20240502 14:23:31.099437 5733 consensus_queue.cc:237] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } }
I20240502 14:23:31.102083 5864 raft_consensus.cc:1261] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Refusing update from remote peer 1dbf482780eb47e8933983e4f0fd0444: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:23:31.103428 7931 consensus_queue.cc:1035] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 14:23:31.104164 6123 raft_consensus.cc:1261] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Refusing update from remote peer 1dbf482780eb47e8933983e4f0fd0444: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:23:31.105674 7094 consensus_queue.cc:1035] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [LEADER]: Connected to new peer: Peer: permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:23:31.106604 7094 raft_consensus.cc:2932] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } } }
I20240502 14:23:31.110610 5864 raft_consensus.cc:2932] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } } }
I20240502 14:23:31.114682 6123 raft_consensus.cc:2932] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } } }
I20240502 14:23:31.116142 5576 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a6647f23da6c4430a9f84e378dd345a4 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:23:31.116015 5589 catalog_manager.cc:5484] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081 reported cstate change: config changed from index 4 to 5, NON_VOTER 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) added. New cstate: current_term: 1 leader_uuid: "1dbf482780eb47e8933983e4f0fd0444" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } } }
W20240502 14:23:31.120932 5669 consensus_peers.cc:485] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 -> Peer 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70:40027): Couldn't send request to peer 806206bb049040bf8db48a7d8d3840c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6647f23da6c4430a9f84e378dd345a4. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:23:31.132642 5733 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a6647f23da6c4430a9f84e378dd345a4"
dest_uuid: "1dbf482780eb47e8933983e4f0fd0444"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41168
I20240502 14:23:31.132900 5733 raft_consensus.cc:595] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:23:31.213948 5713 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:23:31.213953 7484 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:31.237015 7614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:31.247207 5973 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:23:31.256485 5844 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:23:31.262449 6103 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:23:31.262853 7222 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:31.285591 7354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:31.611471 7989 ts_tablet_manager.cc:902] T a6647f23da6c4430a9f84e378dd345a4 P 806206bb049040bf8db48a7d8d3840c1: Initiating tablet copy from peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831)
I20240502 14:23:31.613318 7989 tablet_copy_client.cc:308] T a6647f23da6c4430a9f84e378dd345a4 P 806206bb049040bf8db48a7d8d3840c1: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.65:44831
I20240502 14:23:31.620492 5754 tablet_copy_service.cc:140] P 1dbf482780eb47e8933983e4f0fd0444: Received BeginTabletCopySession request for tablet a6647f23da6c4430a9f84e378dd345a4 from peer 806206bb049040bf8db48a7d8d3840c1 ({username='slave'} at 127.5.107.70:53947)
I20240502 14:23:31.620637 5754 tablet_copy_service.cc:161] P 1dbf482780eb47e8933983e4f0fd0444: Beginning new tablet copy session on tablet a6647f23da6c4430a9f84e378dd345a4 from peer 806206bb049040bf8db48a7d8d3840c1 at {username='slave'} at 127.5.107.70:53947: session id = 806206bb049040bf8db48a7d8d3840c1-a6647f23da6c4430a9f84e378dd345a4
I20240502 14:23:31.623041 5754 tablet_copy_source_session.cc:205] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:23:31.626976 7989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6647f23da6c4430a9f84e378dd345a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:31.636194 7989 tablet_copy_client.cc:783] T a6647f23da6c4430a9f84e378dd345a4 P 806206bb049040bf8db48a7d8d3840c1: tablet copy: Starting download of 0 data blocks...
I20240502 14:23:31.636531 7989 tablet_copy_client.cc:647] T a6647f23da6c4430a9f84e378dd345a4 P 806206bb049040bf8db48a7d8d3840c1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:23:31.638845 7993 tablet_copy_client.cc:1002] T a6647f23da6c4430a9f84e378dd345a4 P 806206bb049040bf8db48a7d8d3840c1: tablet copy: Injecting latency into file download: 1500
I20240502 14:23:31.686553 7094 raft_consensus.cc:979] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444: : Instructing follower 333acff728274715a9816c286921ddd0 to start an election
I20240502 14:23:31.686718 7094 raft_consensus.cc:1067] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 LEADER]: Signalling peer 333acff728274715a9816c286921ddd0 to start an election
I20240502 14:23:31.687580 6123 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a6647f23da6c4430a9f84e378dd345a4"
dest_uuid: "333acff728274715a9816c286921ddd0"
from {username='slave'} at 127.5.107.65:46819
I20240502 14:23:31.687939 6123 raft_consensus.cc:482] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:23:31.688063 6123 raft_consensus.cc:3037] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:23:31.690938 6123 raft_consensus.cc:504] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } }
I20240502 14:23:31.692176 5733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6647f23da6c4430a9f84e378dd345a4" candidate_uuid: "333acff728274715a9816c286921ddd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "1dbf482780eb47e8933983e4f0fd0444"
I20240502 14:23:31.692268 6123 leader_election.cc:288] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 2 election: Requested vote from peers 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831), 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015)
I20240502 14:23:31.692384 5733 raft_consensus.cc:3032] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:23:31.692461 5733 raft_consensus.cc:729] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1dbf482780eb47e8933983e4f0fd0444, State: Running, Role: LEADER
I20240502 14:23:31.692392 5864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6647f23da6c4430a9f84e378dd345a4" candidate_uuid: "333acff728274715a9816c286921ddd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081"
I20240502 14:23:31.692586 5864 raft_consensus.cc:3037] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:23:31.692759 5733 consensus_queue.cc:260] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } }
I20240502 14:23:31.693217 5733 raft_consensus.cc:3037] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:23:31.694998 5864 raft_consensus.cc:2454] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 333acff728274715a9816c286921ddd0 in term 2.
I20240502 14:23:31.695411 6058 leader_election.cc:302] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 333acff728274715a9816c286921ddd0, 83e62d4e5f8a4316bc1bcb8c18458081; no voters:
I20240502 14:23:31.695575 5733 raft_consensus.cc:2454] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 333acff728274715a9816c286921ddd0 in term 2.
I20240502 14:23:31.695709 7746 raft_consensus.cc:2781] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:23:31.696098 7746 raft_consensus.cc:686] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [term 2 LEADER]: Becoming Leader. State: Replica: 333acff728274715a9816c286921ddd0, State: Running, Role: LEADER
I20240502 14:23:31.696398 7746 consensus_queue.cc:237] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } }
I20240502 14:23:31.699473 5589 catalog_manager.cc:5484] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 reported cstate change: term changed from 1 to 2, leader changed from 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65) to 333acff728274715a9816c286921ddd0 (127.5.107.68). New cstate: current_term: 2 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:23:31.750819 7983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:23:31.751219 7983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:23:31.760792 7983 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:23:31.907018 5973 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:23:31.908418 5713 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:23:31.916584 6103 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:23:31.936622 7354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:31.984337 7222 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:31.997612 7614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:32.007246 7484 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:32.024755 5844 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Runtime error: ksck discovered errors
I20240502 14:23:32.115453 5733 raft_consensus.cc:1261] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [term 2 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:23:32.116042 7746 consensus_queue.cc:1035] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:23:32.122628 5864 raft_consensus.cc:1261] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:23:32.123575 7746 consensus_queue.cc:1035] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 14:23:32.160126 6059 consensus_peers.cc:485] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 -> Peer 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70:40027): Couldn't send request to peer 806206bb049040bf8db48a7d8d3840c1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:23:32.194455 7747 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 5652
I20240502 14:23:32.201373 5774 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:23:32.329838 7747 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 5652
W20240502 14:23:32.352023 6058 connection.cc:535] server connection from 127.5.107.65:46819 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:23:32.352475 6057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.107.65:44831 (error 108)
W20240502 14:23:32.353988 6057 consensus_peers.cc:485] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:23:32.521421 5927 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
W20240502 14:23:32.522702 5927 consensus_peers.cc:485] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:23:32.615907 5927 consensus_peers.cc:485] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:23:32.626363 6057 consensus_peers.cc:485] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:23:32.653753 5797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
W20240502 14:23:32.654973 5797 consensus_peers.cc:485] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:23:32.687868 6057 consensus_peers.cc:485] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:23:32.749779 6057 consensus_peers.cc:485] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:23:32.756641 6057 consensus_peers.cc:485] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:23:33.139341 7989 tablet_copy_client.cc:523] T a6647f23da6c4430a9f84e378dd345a4 P 806206bb049040bf8db48a7d8d3840c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:23:33.142668 7989 tablet_bootstrap.cc:492] T a6647f23da6c4430a9f84e378dd345a4 P 806206bb049040bf8db48a7d8d3840c1: Bootstrap starting.
I20240502 14:23:33.152554 7989 log.cc:824] T a6647f23da6c4430a9f84e378dd345a4 P 806206bb049040bf8db48a7d8d3840c1: Log is configured to *not* fsync() on all Append() calls
I20240502 14:23:33.181958 7989 tablet_bootstrap.cc:492] T a6647f23da6c4430a9f84e378dd345a4 P 806206bb049040bf8db48a7d8d3840c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:23:33.182417 7989 tablet_bootstrap.cc:492] T a6647f23da6c4430a9f84e378dd345a4 P 806206bb049040bf8db48a7d8d3840c1: Bootstrap complete.
I20240502 14:23:33.182767 7989 ts_tablet_manager.cc:1367] T a6647f23da6c4430a9f84e378dd345a4 P 806206bb049040bf8db48a7d8d3840c1: Time spent bootstrapping tablet: real 0.040s user 0.039s sys 0.001s
I20240502 14:23:33.186945 7989 raft_consensus.cc:348] T a6647f23da6c4430a9f84e378dd345a4 P 806206bb049040bf8db48a7d8d3840c1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } }
I20240502 14:23:33.187294 7989 raft_consensus.cc:729] T a6647f23da6c4430a9f84e378dd345a4 P 806206bb049040bf8db48a7d8d3840c1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 806206bb049040bf8db48a7d8d3840c1, State: Initialized, Role: LEARNER
I20240502 14:23:33.187753 7989 consensus_queue.cc:260] T a6647f23da6c4430a9f84e378dd345a4 P 806206bb049040bf8db48a7d8d3840c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } }
I20240502 14:23:33.189195 7989 ts_tablet_manager.cc:1398] T a6647f23da6c4430a9f84e378dd345a4 P 806206bb049040bf8db48a7d8d3840c1: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
W20240502 14:23:33.190623 7989 tablet_copy_client.cc:1108] T a6647f23da6c4430a9f84e378dd345a4 P 806206bb049040bf8db48a7d8d3840c1: tablet copy: Unable to close tablet copy session: Network error: failure ending tablet copy session: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
I20240502 14:23:33.199008 6739 raft_consensus.cc:482] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1dbf482780eb47e8933983e4f0fd0444)
I20240502 14:23:33.199149 6739 raft_consensus.cc:504] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:33.199640 6739 leader_election.cc:288] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831), 333acff728274715a9816c286921ddd0 (127.5.107.68:45913)
I20240502 14:23:33.200055 6123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f9ebd715dc4f96901b60796ea846ad" candidate_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "333acff728274715a9816c286921ddd0" is_pre_election: true
W20240502 14:23:33.201287 5797 leader_election.cc:334] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
I20240502 14:23:33.201408 5797 leader_election.cc:302] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83e62d4e5f8a4316bc1bcb8c18458081; no voters: 1dbf482780eb47e8933983e4f0fd0444, 333acff728274715a9816c286921ddd0
I20240502 14:23:33.201658 6739 raft_consensus.cc:2726] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:23:33.289744 8103 raft_consensus.cc:482] T 0697cc9c0d26419083f71a411b2699ed P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1dbf482780eb47e8933983e4f0fd0444)
I20240502 14:23:33.289940 8103 raft_consensus.cc:504] T 0697cc9c0d26419083f71a411b2699ed P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:23:33.290647 8103 leader_election.cc:288] T 0697cc9c0d26419083f71a411b2699ed P 181b466de0084fcbace0ca327fb1cbef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015), 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831)
I20240502 14:23:33.291184 5864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0697cc9c0d26419083f71a411b2699ed" candidate_uuid: "181b466de0084fcbace0ca327fb1cbef" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" is_pre_election: true
W20240502 14:23:33.292140 5927 leader_election.cc:334] T 0697cc9c0d26419083f71a411b2699ed P 181b466de0084fcbace0ca327fb1cbef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
I20240502 14:23:33.292268 5927 leader_election.cc:302] T 0697cc9c0d26419083f71a411b2699ed P 181b466de0084fcbace0ca327fb1cbef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 181b466de0084fcbace0ca327fb1cbef; no voters: 1dbf482780eb47e8933983e4f0fd0444, 83e62d4e5f8a4316bc1bcb8c18458081
I20240502 14:23:33.292563 8103 raft_consensus.cc:2726] T 0697cc9c0d26419083f71a411b2699ed P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:23:33.433295 8105 raft_consensus.cc:482] T 244218906a51489dab769122731b1ec2 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1dbf482780eb47e8933983e4f0fd0444)
I20240502 14:23:33.433460 8105 raft_consensus.cc:504] T 244218906a51489dab769122731b1ec2 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:33.434092 8105 leader_election.cc:288] T 244218906a51489dab769122731b1ec2 P 83e62d4e5f8a4316bc1bcb8c18458081 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 333acff728274715a9816c286921ddd0 (127.5.107.68:45913), 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831)
I20240502 14:23:33.434468 6123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "244218906a51489dab769122731b1ec2" candidate_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "333acff728274715a9816c286921ddd0" is_pre_election: true
W20240502 14:23:33.435199 5797 leader_election.cc:334] T 244218906a51489dab769122731b1ec2 P 83e62d4e5f8a4316bc1bcb8c18458081 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
I20240502 14:23:33.435317 5797 leader_election.cc:302] T 244218906a51489dab769122731b1ec2 P 83e62d4e5f8a4316bc1bcb8c18458081 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83e62d4e5f8a4316bc1bcb8c18458081; no voters: 1dbf482780eb47e8933983e4f0fd0444, 333acff728274715a9816c286921ddd0
I20240502 14:23:33.435506 8105 raft_consensus.cc:2726] T 244218906a51489dab769122731b1ec2 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:23:33.443142 8085 raft_consensus.cc:482] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1dbf482780eb47e8933983e4f0fd0444)
I20240502 14:23:33.443312 8085 raft_consensus.cc:504] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:33.443810 8085 leader_election.cc:288] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831), 181b466de0084fcbace0ca327fb1cbef (127.5.107.67:41517)
I20240502 14:23:33.444293 5993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca9200e5c264c27833d7a4a9122c4d9" candidate_uuid: "333acff728274715a9816c286921ddd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "181b466de0084fcbace0ca327fb1cbef" is_pre_election: true
W20240502 14:23:33.445000 6057 leader_election.cc:334] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
I20240502 14:23:33.445111 6057 leader_election.cc:302] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 333acff728274715a9816c286921ddd0; no voters: 181b466de0084fcbace0ca327fb1cbef, 1dbf482780eb47e8933983e4f0fd0444
I20240502 14:23:33.445391 8085 raft_consensus.cc:2726] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:23:33.446413 8105 raft_consensus.cc:482] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1dbf482780eb47e8933983e4f0fd0444)
I20240502 14:23:33.446530 8105 raft_consensus.cc:504] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:23:33.447041 8105 leader_election.cc:288] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 181b466de0084fcbace0ca327fb1cbef (127.5.107.67:41517), 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831)
I20240502 14:23:33.447407 5993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0697cc9c0d26419083f71a411b2699ed" candidate_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "181b466de0084fcbace0ca327fb1cbef" is_pre_election: true
I20240502 14:23:33.447613 5993 raft_consensus.cc:2454] T 0697cc9c0d26419083f71a411b2699ed P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e62d4e5f8a4316bc1bcb8c18458081 in term 1.
I20240502 14:23:33.447973 5799 leader_election.cc:302] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 181b466de0084fcbace0ca327fb1cbef, 83e62d4e5f8a4316bc1bcb8c18458081; no voters:
I20240502 14:23:33.448206 8105 raft_consensus.cc:2781] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240502 14:23:33.448288 5797 leader_election.cc:334] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
I20240502 14:23:33.448312 8105 raft_consensus.cc:482] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1dbf482780eb47e8933983e4f0fd0444)
I20240502 14:23:33.448403 8105 raft_consensus.cc:3037] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:23:33.450152 8105 raft_consensus.cc:504] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:23:33.450691 8105 leader_election.cc:288] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 [CANDIDATE]: Term 2 election: Requested vote from peers 181b466de0084fcbace0ca327fb1cbef (127.5.107.67:41517), 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831)
I20240502 14:23:33.450956 5993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0697cc9c0d26419083f71a411b2699ed" candidate_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "181b466de0084fcbace0ca327fb1cbef"
I20240502 14:23:33.451112 5993 raft_consensus.cc:3037] T 0697cc9c0d26419083f71a411b2699ed P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:23:33.451772 5797 leader_election.cc:334] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
I20240502 14:23:33.452908 5993 raft_consensus.cc:2454] T 0697cc9c0d26419083f71a411b2699ed P 181b466de0084fcbace0ca327fb1cbef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e62d4e5f8a4316bc1bcb8c18458081 in term 2.
I20240502 14:23:33.453270 5799 leader_election.cc:302] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 181b466de0084fcbace0ca327fb1cbef, 83e62d4e5f8a4316bc1bcb8c18458081; no voters: 1dbf482780eb47e8933983e4f0fd0444
I20240502 14:23:33.453518 8105 raft_consensus.cc:2781] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:23:33.453663 8105 raft_consensus.cc:686] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 LEADER]: Becoming Leader. State: Replica: 83e62d4e5f8a4316bc1bcb8c18458081, State: Running, Role: LEADER
I20240502 14:23:33.453948 8105 consensus_queue.cc:237] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:23:33.456230 5586 catalog_manager.cc:5484] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 reported cstate change: term changed from 1 to 2, leader changed from 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65) to 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66). New cstate: current_term: 2 leader_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20240502 14:23:33.476876 8085 raft_consensus.cc:482] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1dbf482780eb47e8933983e4f0fd0444)
I20240502 14:23:33.477048 8085 raft_consensus.cc:504] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:33.477627 8085 leader_election.cc:288] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831), 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015)
I20240502 14:23:33.478132 5864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f9ebd715dc4f96901b60796ea846ad" candidate_uuid: "333acff728274715a9816c286921ddd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" is_pre_election: true
I20240502 14:23:33.478336 5864 raft_consensus.cc:2454] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 333acff728274715a9816c286921ddd0 in term 1.
I20240502 14:23:33.478685 6058 leader_election.cc:302] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 333acff728274715a9816c286921ddd0, 83e62d4e5f8a4316bc1bcb8c18458081; no voters:
I20240502 14:23:33.478937 8085 raft_consensus.cc:2781] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240502 14:23:33.478917 6057 leader_election.cc:334] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
I20240502 14:23:33.479084 8085 raft_consensus.cc:482] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1dbf482780eb47e8933983e4f0fd0444)
I20240502 14:23:33.479173 8085 raft_consensus.cc:3037] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:23:33.480882 8085 raft_consensus.cc:504] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:33.481410 8085 leader_election.cc:288] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 2 election: Requested vote from peers 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831), 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015)
I20240502 14:23:33.481890 5864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f9ebd715dc4f96901b60796ea846ad" candidate_uuid: "333acff728274715a9816c286921ddd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081"
I20240502 14:23:33.482101 5864 raft_consensus.cc:3037] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:23:33.482709 6057 leader_election.cc:334] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
I20240502 14:23:33.483899 5864 raft_consensus.cc:2454] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 333acff728274715a9816c286921ddd0 in term 2.
I20240502 14:23:33.484210 6058 leader_election.cc:302] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 333acff728274715a9816c286921ddd0, 83e62d4e5f8a4316bc1bcb8c18458081; no voters: 1dbf482780eb47e8933983e4f0fd0444
I20240502 14:23:33.484445 8085 raft_consensus.cc:2781] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:23:33.484558 8085 raft_consensus.cc:686] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [term 2 LEADER]: Becoming Leader. State: Replica: 333acff728274715a9816c286921ddd0, State: Running, Role: LEADER
I20240502 14:23:33.484787 8085 consensus_queue.cc:237] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:33.487223 5586 catalog_manager.cc:5484] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 reported cstate change: term changed from 1 to 2, leader changed from 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65) to 333acff728274715a9816c286921ddd0 (127.5.107.68). New cstate: current_term: 2 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } health_report { overall_health: UNKNOWN } } }
I20240502 14:23:33.507478 8103 raft_consensus.cc:482] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1dbf482780eb47e8933983e4f0fd0444)
I20240502 14:23:33.507637 8103 raft_consensus.cc:504] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:33.508277 8103 leader_election.cc:288] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 333acff728274715a9816c286921ddd0 (127.5.107.68:45913), 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831)
I20240502 14:23:33.508733 6123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca9200e5c264c27833d7a4a9122c4d9" candidate_uuid: "181b466de0084fcbace0ca327fb1cbef" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "333acff728274715a9816c286921ddd0" is_pre_election: true
I20240502 14:23:33.508919 6123 raft_consensus.cc:2454] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 181b466de0084fcbace0ca327fb1cbef in term 1.
I20240502 14:23:33.509333 5929 leader_election.cc:302] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 181b466de0084fcbace0ca327fb1cbef, 333acff728274715a9816c286921ddd0; no voters:
I20240502 14:23:33.509584 8103 raft_consensus.cc:2781] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240502 14:23:33.509601 5927 leader_election.cc:334] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
I20240502 14:23:33.509716 8103 raft_consensus.cc:482] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1dbf482780eb47e8933983e4f0fd0444)
I20240502 14:23:33.509819 8103 raft_consensus.cc:3037] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:23:33.511519 8103 raft_consensus.cc:504] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:33.511943 8103 leader_election.cc:288] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [CANDIDATE]: Term 2 election: Requested vote from peers 333acff728274715a9816c286921ddd0 (127.5.107.68:45913), 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831)
I20240502 14:23:33.512290 6123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca9200e5c264c27833d7a4a9122c4d9" candidate_uuid: "181b466de0084fcbace0ca327fb1cbef" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "333acff728274715a9816c286921ddd0"
I20240502 14:23:33.512452 6123 raft_consensus.cc:3037] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:23:33.512985 5927 leader_election.cc:334] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
I20240502 14:23:33.514165 6123 raft_consensus.cc:2454] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 181b466de0084fcbace0ca327fb1cbef in term 2.
I20240502 14:23:33.514444 5929 leader_election.cc:302] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 181b466de0084fcbace0ca327fb1cbef, 333acff728274715a9816c286921ddd0; no voters: 1dbf482780eb47e8933983e4f0fd0444
I20240502 14:23:33.514696 8103 raft_consensus.cc:2781] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:23:33.514822 8103 raft_consensus.cc:686] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [term 2 LEADER]: Becoming Leader. State: Replica: 181b466de0084fcbace0ca327fb1cbef, State: Running, Role: LEADER
I20240502 14:23:33.515072 8103 consensus_queue.cc:237] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:33.517405 5586 catalog_manager.cc:5484] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef reported cstate change: term changed from 1 to 2, leader changed from 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65) to 181b466de0084fcbace0ca327fb1cbef (127.5.107.67). New cstate: current_term: 2 leader_uuid: "181b466de0084fcbace0ca327fb1cbef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } health_report { overall_health: HEALTHY } } }
I20240502 14:23:33.621253 7374 raft_consensus.cc:3037] T a6647f23da6c4430a9f84e378dd345a4 P 806206bb049040bf8db48a7d8d3840c1 [term 1 LEARNER]: Advancing to term 2
I20240502 14:23:33.830765 7996 raft_consensus.cc:482] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1dbf482780eb47e8933983e4f0fd0444)
I20240502 14:23:33.830948 7996 raft_consensus.cc:504] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:33.831678 7996 leader_election.cc:288] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831), 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015)
I20240502 14:23:33.832293 5864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "244218906a51489dab769122731b1ec2" candidate_uuid: "333acff728274715a9816c286921ddd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" is_pre_election: true
I20240502 14:23:33.832543 5864 raft_consensus.cc:2454] T 244218906a51489dab769122731b1ec2 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 333acff728274715a9816c286921ddd0 in term 1.
I20240502 14:23:33.833000 6058 leader_election.cc:302] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 333acff728274715a9816c286921ddd0, 83e62d4e5f8a4316bc1bcb8c18458081; no voters:
W20240502 14:23:33.833228 6057 leader_election.cc:334] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
I20240502 14:23:33.833297 7996 raft_consensus.cc:2781] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:23:33.833420 7996 raft_consensus.cc:482] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1dbf482780eb47e8933983e4f0fd0444)
I20240502 14:23:33.833508 7996 raft_consensus.cc:3037] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:23:33.835230 7996 raft_consensus.cc:504] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:33.835662 7996 leader_election.cc:288] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 2 election: Requested vote from peers 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831), 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015)
I20240502 14:23:33.836192 5864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "244218906a51489dab769122731b1ec2" candidate_uuid: "333acff728274715a9816c286921ddd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081"
I20240502 14:23:33.836371 5864 raft_consensus.cc:3037] T 244218906a51489dab769122731b1ec2 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:23:33.836697 6057 leader_election.cc:334] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
I20240502 14:23:33.838395 5864 raft_consensus.cc:2454] T 244218906a51489dab769122731b1ec2 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 333acff728274715a9816c286921ddd0 in term 2.
I20240502 14:23:33.838716 6058 leader_election.cc:302] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 333acff728274715a9816c286921ddd0, 83e62d4e5f8a4316bc1bcb8c18458081; no voters: 1dbf482780eb47e8933983e4f0fd0444
I20240502 14:23:33.838974 7996 raft_consensus.cc:2781] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:23:33.839110 7996 raft_consensus.cc:686] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 [term 2 LEADER]: Becoming Leader. State: Replica: 333acff728274715a9816c286921ddd0, State: Running, Role: LEADER
I20240502 14:23:33.839320 7996 consensus_queue.cc:237] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:33.842203 5586 catalog_manager.cc:5484] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 reported cstate change: term changed from 1 to 2, leader changed from 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65) to 333acff728274715a9816c286921ddd0 (127.5.107.68). New cstate: current_term: 2 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } health_report { overall_health: UNKNOWN } } }
I20240502 14:23:33.858237 5993 raft_consensus.cc:1261] T 0697cc9c0d26419083f71a411b2699ed P 181b466de0084fcbace0ca327fb1cbef [term 2 FOLLOWER]: Refusing update from remote peer 83e62d4e5f8a4316bc1bcb8c18458081: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:23:33.858690 8105 consensus_queue.cc:1035] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:23:33.860605 5797 consensus_peers.cc:485] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:23:33.896704 6123 raft_consensus.cc:1261] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Refusing update from remote peer 181b466de0084fcbace0ca327fb1cbef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:23:33.897337 8103 consensus_queue.cc:1035] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Connected to new peer: Peer: permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:23:33.901887 5927 consensus_peers.cc:485] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:23:33.965925 5864 raft_consensus.cc:1261] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:23:33.966384 7996 consensus_queue.cc:1035] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:23:33.970853 6057 consensus_peers.cc:485] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:23:34.216599 8143 raft_consensus.cc:1050] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0: attempting to promote NON_VOTER 806206bb049040bf8db48a7d8d3840c1 to VOTER
I20240502 14:23:34.217950 8143 consensus_queue.cc:237] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } }
I20240502 14:23:34.220135 7374 raft_consensus.cc:1261] T a6647f23da6c4430a9f84e378dd345a4 P 806206bb049040bf8db48a7d8d3840c1 [term 2 LEARNER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:23:34.220511 5864 raft_consensus.cc:1261] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:23:34.220707 8143 consensus_queue.cc:1035] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:23:34.221274 8150 consensus_queue.cc:1035] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:23:34.221836 6057 consensus_peers.cc:485] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:23:34.223958 8143 raft_consensus.cc:2932] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:23:34.224519 5864 raft_consensus.cc:2932] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:23:34.227319 7374 raft_consensus.cc:2932] T a6647f23da6c4430a9f84e378dd345a4 P 806206bb049040bf8db48a7d8d3840c1 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:23:34.228507 5586 catalog_manager.cc:5484] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 reported cstate change: config changed from index 5 to 7, 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:23:34.232766 6123 consensus_queue.cc:237] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } }
I20240502 14:23:34.234375 5864 raft_consensus.cc:1261] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240502 14:23:34.234675 6057 consensus_peers.cc:485] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:23:34.234838 7374 raft_consensus.cc:1261] T a6647f23da6c4430a9f84e378dd345a4 P 806206bb049040bf8db48a7d8d3840c1 [term 2 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:23:34.235266 8150 consensus_queue.cc:1035] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:23:34.235596 8150 consensus_queue.cc:1035] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:23:34.237672 8085 raft_consensus.cc:2932] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:23:34.238395 7374 raft_consensus.cc:2932] T a6647f23da6c4430a9f84e378dd345a4 P 806206bb049040bf8db48a7d8d3840c1 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:23:34.238395 5864 raft_consensus.cc:2932] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:23:34.241346 5578 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a6647f23da6c4430a9f84e378dd345a4 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:23:34.242430 5586 catalog_manager.cc:5484] T a6647f23da6c4430a9f84e378dd345a4 P 806206bb049040bf8db48a7d8d3840c1 reported cstate change: config changed from index 7 to 8, VOTER 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65) evicted. New cstate: current_term: 2 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
W20240502 14:23:34.249892 5576 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
W20240502 14:23:34.251211 5576 catalog_manager.cc:4573] TS 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a6647f23da6c4430a9f84e378dd345a4: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
I20240502 14:23:34.327944 5864 raft_consensus.cc:1261] T 244218906a51489dab769122731b1ec2 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:23:34.328506 8143 consensus_queue.cc:1035] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:23:34.572911 6123 consensus_queue.cc:237] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:34.574928 5864 raft_consensus.cc:1261] T 7d944a63fdf24ecd8fadbad3b620af0a P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:23:34.575511 8150 consensus_queue.cc:1035] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:23:34.575840 6057 consensus_peers.cc:485] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:23:34.578051 8143 raft_consensus.cc:2932] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } }
I20240502 14:23:34.578603 5864 raft_consensus.cc:2932] T 7d944a63fdf24ecd8fadbad3b620af0a P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } }
I20240502 14:23:34.582792 5586 catalog_manager.cc:5484] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } health_report { overall_health: HEALTHY } } }
I20240502 14:23:34.587402 6123 consensus_queue.cc:237] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } }
W20240502 14:23:34.589946 6059 consensus_peers.cc:485] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 -> Peer 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70:40027): Couldn't send request to peer 806206bb049040bf8db48a7d8d3840c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d944a63fdf24ecd8fadbad3b620af0a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:23:34.593901 5864 raft_consensus.cc:1261] T 7d944a63fdf24ecd8fadbad3b620af0a P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:23:34.594395 8143 consensus_queue.cc:1035] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:23:34.594367 6057 consensus_peers.cc:485] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:23:34.597215 8085 raft_consensus.cc:2932] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } } }
I20240502 14:23:34.601778 5863 raft_consensus.cc:2932] T 7d944a63fdf24ecd8fadbad3b620af0a P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } } }
I20240502 14:23:34.607048 5586 catalog_manager.cc:5484] T 7d944a63fdf24ecd8fadbad3b620af0a P 83e62d4e5f8a4316bc1bcb8c18458081 reported cstate change: config changed from index 4 to 5, NON_VOTER 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) added. New cstate: current_term: 1 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } } }
I20240502 14:23:34.608115 5578 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7d944a63fdf24ecd8fadbad3b620af0a with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240502 14:23:34.852155 5927 consensus_peers.cc:485] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:23:34.924253 6057 consensus_peers.cc:485] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:23:35.085880 6057 consensus_peers.cc:485] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:23:35.106937 8183 ts_tablet_manager.cc:902] T 7d944a63fdf24ecd8fadbad3b620af0a P 806206bb049040bf8db48a7d8d3840c1: Initiating tablet copy from peer 333acff728274715a9816c286921ddd0 (127.5.107.68:45913)
I20240502 14:23:35.108304 8183 tablet_copy_client.cc:308] T 7d944a63fdf24ecd8fadbad3b620af0a P 806206bb049040bf8db48a7d8d3840c1: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.68:45913
I20240502 14:23:35.113792 6144 tablet_copy_service.cc:140] P 333acff728274715a9816c286921ddd0: Received BeginTabletCopySession request for tablet 7d944a63fdf24ecd8fadbad3b620af0a from peer 806206bb049040bf8db48a7d8d3840c1 ({username='slave'} at 127.5.107.70:36379)
I20240502 14:23:35.113968 6144 tablet_copy_service.cc:161] P 333acff728274715a9816c286921ddd0: Beginning new tablet copy session on tablet 7d944a63fdf24ecd8fadbad3b620af0a from peer 806206bb049040bf8db48a7d8d3840c1 at {username='slave'} at 127.5.107.70:36379: session id = 806206bb049040bf8db48a7d8d3840c1-7d944a63fdf24ecd8fadbad3b620af0a
I20240502 14:23:35.116400 6144 tablet_copy_source_session.cc:205] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:23:35.118070 8183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d944a63fdf24ecd8fadbad3b620af0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:35.122174 8183 tablet_copy_client.cc:783] T 7d944a63fdf24ecd8fadbad3b620af0a P 806206bb049040bf8db48a7d8d3840c1: tablet copy: Starting download of 0 data blocks...
I20240502 14:23:35.122416 8183 tablet_copy_client.cc:647] T 7d944a63fdf24ecd8fadbad3b620af0a P 806206bb049040bf8db48a7d8d3840c1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:23:35.123976 8187 tablet_copy_client.cc:1002] T 7d944a63fdf24ecd8fadbad3b620af0a P 806206bb049040bf8db48a7d8d3840c1: tablet copy: Injecting latency into file download: 1500
W20240502 14:23:35.146268 6057 consensus_peers.cc:485] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:23:35.276541 5797 consensus_peers.cc:485] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:23:35.287737 5927 consensus_peers.cc:485] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:23:35.462395 5576 catalog_manager.cc:4573] TS 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a6647f23da6c4430a9f84e378dd345a4: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
W20240502 14:23:35.915972 5797 consensus_peers.cc:485] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:23:36.291384 5927 consensus_peers.cc:485] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:23:36.409564 6057 consensus_peers.cc:485] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:23:36.519141 6057 consensus_peers.cc:485] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:23:36.531368 5576 catalog_manager.cc:4573] TS 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a6647f23da6c4430a9f84e378dd345a4: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
I20240502 14:23:36.624418 8183 tablet_copy_client.cc:523] T 7d944a63fdf24ecd8fadbad3b620af0a P 806206bb049040bf8db48a7d8d3840c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:23:36.626832 8183 tablet_bootstrap.cc:492] T 7d944a63fdf24ecd8fadbad3b620af0a P 806206bb049040bf8db48a7d8d3840c1: Bootstrap starting.
I20240502 14:23:36.653491 8183 tablet_bootstrap.cc:492] T 7d944a63fdf24ecd8fadbad3b620af0a P 806206bb049040bf8db48a7d8d3840c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:23:36.653805 8183 tablet_bootstrap.cc:492] T 7d944a63fdf24ecd8fadbad3b620af0a P 806206bb049040bf8db48a7d8d3840c1: Bootstrap complete.
I20240502 14:23:36.654026 8183 ts_tablet_manager.cc:1367] T 7d944a63fdf24ecd8fadbad3b620af0a P 806206bb049040bf8db48a7d8d3840c1: Time spent bootstrapping tablet: real 0.027s user 0.024s sys 0.004s
I20240502 14:23:36.654700 8183 raft_consensus.cc:348] T 7d944a63fdf24ecd8fadbad3b620af0a P 806206bb049040bf8db48a7d8d3840c1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } }
I20240502 14:23:36.654893 8183 raft_consensus.cc:729] T 7d944a63fdf24ecd8fadbad3b620af0a P 806206bb049040bf8db48a7d8d3840c1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 806206bb049040bf8db48a7d8d3840c1, State: Initialized, Role: LEARNER
I20240502 14:23:36.655054 8183 consensus_queue.cc:260] T 7d944a63fdf24ecd8fadbad3b620af0a P 806206bb049040bf8db48a7d8d3840c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } }
I20240502 14:23:36.655587 8183 ts_tablet_manager.cc:1398] T 7d944a63fdf24ecd8fadbad3b620af0a P 806206bb049040bf8db48a7d8d3840c1: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:23:36.656234 6144 tablet_copy_service.cc:342] P 333acff728274715a9816c286921ddd0: Request end of tablet copy session 806206bb049040bf8db48a7d8d3840c1-7d944a63fdf24ecd8fadbad3b620af0a received from {username='slave'} at 127.5.107.70:36379
I20240502 14:23:36.656376 6144 tablet_copy_service.cc:433] P 333acff728274715a9816c286921ddd0: ending tablet copy session 806206bb049040bf8db48a7d8d3840c1-7d944a63fdf24ecd8fadbad3b620af0a on tablet 7d944a63fdf24ecd8fadbad3b620af0a with peer 806206bb049040bf8db48a7d8d3840c1
W20240502 14:23:37.015288 6057 consensus_peers.cc:485] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:23:37.142413 5927 consensus_peers.cc:485] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:23:37.213066 7374 raft_consensus.cc:1203] T 7d944a63fdf24ecd8fadbad3b620af0a P 806206bb049040bf8db48a7d8d3840c1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240502 14:23:37.359578 5927 consensus_peers.cc:485] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:23:37.481264 6057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240502 14:23:37.494936 6057 consensus_peers.cc:485] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:23:37.651033 6057 consensus_peers.cc:485] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:23:37.672472 5927 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240502 14:23:37.753741 6057 consensus_peers.cc:485] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:23:37.756057 8265 raft_consensus.cc:1050] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0: attempting to promote NON_VOTER 806206bb049040bf8db48a7d8d3840c1 to VOTER
I20240502 14:23:37.756947 8265 consensus_queue.cc:237] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } }
I20240502 14:23:37.759456 5863 raft_consensus.cc:1261] T 7d944a63fdf24ecd8fadbad3b620af0a P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:23:37.759964 7374 raft_consensus.cc:1261] T 7d944a63fdf24ecd8fadbad3b620af0a P 806206bb049040bf8db48a7d8d3840c1 [term 1 LEARNER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 14:23:37.760224 6057 consensus_peers.cc:485] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:23:37.760568 8267 consensus_queue.cc:1035] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:23:37.760929 8265 consensus_queue.cc:1035] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:23:37.763440 8267 raft_consensus.cc:2932] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:23:37.763937 7374 raft_consensus.cc:2932] T 7d944a63fdf24ecd8fadbad3b620af0a P 806206bb049040bf8db48a7d8d3840c1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:23:37.766561 5863 raft_consensus.cc:2932] T 7d944a63fdf24ecd8fadbad3b620af0a P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:23:37.768105 5586 catalog_manager.cc:5484] T 7d944a63fdf24ecd8fadbad3b620af0a P 806206bb049040bf8db48a7d8d3840c1 reported cstate change: config changed from index 5 to 6, 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:23:37.773768 6123 consensus_queue.cc:237] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } }
I20240502 14:23:37.775133 5863 raft_consensus.cc:1261] T 7d944a63fdf24ecd8fadbad3b620af0a P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:23:37.775239 7374 raft_consensus.cc:1261] T 7d944a63fdf24ecd8fadbad3b620af0a P 806206bb049040bf8db48a7d8d3840c1 [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:23:37.775563 8264 consensus_queue.cc:1035] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:23:37.775734 8267 consensus_queue.cc:1035] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:23:37.777628 8267 raft_consensus.cc:2932] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:23:37.778519 5863 raft_consensus.cc:2932] T 7d944a63fdf24ecd8fadbad3b620af0a P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:23:37.780218 5578 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7d944a63fdf24ecd8fadbad3b620af0a with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:23:37.781276 7374 raft_consensus.cc:2932] T 7d944a63fdf24ecd8fadbad3b620af0a P 806206bb049040bf8db48a7d8d3840c1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:23:37.781894 5589 catalog_manager.cc:5484] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 reported cstate change: config changed from index 6 to 7, VOTER 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65) evicted. New cstate: current_term: 1 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 14:23:37.786597 5576 catalog_manager.cc:4573] TS 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7d944a63fdf24ecd8fadbad3b620af0a: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
W20240502 14:23:37.825819 5797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240502 14:23:37.846979 5797 consensus_peers.cc:485] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:23:38.170396 6123 consensus_queue.cc:237] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:23:38.172582 5863 raft_consensus.cc:1261] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:23:38.173115 8267 consensus_queue.cc:1035] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:23:38.173357 6057 consensus_peers.cc:485] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:23:38.175650 8265 raft_consensus.cc:2932] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } }
I20240502 14:23:38.176167 5863 raft_consensus.cc:2932] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } }
I20240502 14:23:38.180457 5586 catalog_manager.cc:5484] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 reported cstate change: config changed from index -1 to 5. New cstate: current_term: 2 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } }
I20240502 14:23:38.188071 6123 consensus_queue.cc:237] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: NON_VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: true } }
I20240502 14:23:38.190550 5863 raft_consensus.cc:1261] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:23:38.191257 8242 consensus_queue.cc:1035] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:23:38.195770 8242 raft_consensus.cc:2932] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER 3416b273466c495298342b6e892015c2 (127.5.107.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: NON_VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: true } } }
I20240502 14:23:38.200075 5864 raft_consensus.cc:2932] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER 3416b273466c495298342b6e892015c2 (127.5.107.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: NON_VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: true } } }
W20240502 14:23:38.202819 6057 consensus_peers.cc:485] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:23:38.205564 5586 catalog_manager.cc:5484] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 reported cstate change: config changed from index 5 to 6, NON_VOTER 3416b273466c495298342b6e892015c2 (127.5.107.69) added. New cstate: current_term: 2 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: NON_VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: true } } }
I20240502 14:23:38.206797 5578 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 25f9ebd715dc4f96901b60796ea846ad with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240502 14:23:38.211114 6059 consensus_peers.cc:485] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 -> Peer 3416b273466c495298342b6e892015c2 (127.5.107.69:37277): Couldn't send request to peer 3416b273466c495298342b6e892015c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25f9ebd715dc4f96901b60796ea846ad. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:23:38.331771 5797 consensus_peers.cc:485] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:23:38.572341 8301 ts_tablet_manager.cc:902] T 25f9ebd715dc4f96901b60796ea846ad P 3416b273466c495298342b6e892015c2: Initiating tablet copy from peer 333acff728274715a9816c286921ddd0 (127.5.107.68:45913)
I20240502 14:23:38.573982 8301 tablet_copy_client.cc:308] T 25f9ebd715dc4f96901b60796ea846ad P 3416b273466c495298342b6e892015c2: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.68:45913
I20240502 14:23:38.579824 6144 tablet_copy_service.cc:140] P 333acff728274715a9816c286921ddd0: Received BeginTabletCopySession request for tablet 25f9ebd715dc4f96901b60796ea846ad from peer 3416b273466c495298342b6e892015c2 ({username='slave'} at 127.5.107.69:60319)
I20240502 14:23:38.579969 6144 tablet_copy_service.cc:161] P 333acff728274715a9816c286921ddd0: Beginning new tablet copy session on tablet 25f9ebd715dc4f96901b60796ea846ad from peer 3416b273466c495298342b6e892015c2 at {username='slave'} at 127.5.107.69:60319: session id = 3416b273466c495298342b6e892015c2-25f9ebd715dc4f96901b60796ea846ad
I20240502 14:23:38.581787 6144 tablet_copy_source_session.cc:205] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:23:38.585790 8301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f9ebd715dc4f96901b60796ea846ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:38.598248 8301 tablet_copy_client.cc:783] T 25f9ebd715dc4f96901b60796ea846ad P 3416b273466c495298342b6e892015c2: tablet copy: Starting download of 0 data blocks...
I20240502 14:23:38.598603 8301 tablet_copy_client.cc:647] T 25f9ebd715dc4f96901b60796ea846ad P 3416b273466c495298342b6e892015c2: tablet copy: Starting download of 1 WAL segments...
I20240502 14:23:38.601769 8305 tablet_copy_client.cc:1002] T 25f9ebd715dc4f96901b60796ea846ad P 3416b273466c495298342b6e892015c2: tablet copy: Injecting latency into file download: 1500
W20240502 14:23:38.662986 5927 consensus_peers.cc:485] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:23:38.927523 5576 catalog_manager.cc:4573] TS 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7d944a63fdf24ecd8fadbad3b620af0a: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
W20240502 14:23:38.990653 6057 consensus_peers.cc:485] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:23:39.488077 5927 consensus_peers.cc:485] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:23:39.992730 5576 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 15 similar messages]
W20240502 14:23:39.993775 5576 catalog_manager.cc:4573] TS 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7d944a63fdf24ecd8fadbad3b620af0a: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
I20240502 14:23:40.102326 8301 tablet_copy_client.cc:523] T 25f9ebd715dc4f96901b60796ea846ad P 3416b273466c495298342b6e892015c2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:23:40.105540 8301 tablet_bootstrap.cc:492] T 25f9ebd715dc4f96901b60796ea846ad P 3416b273466c495298342b6e892015c2: Bootstrap starting.
W20240502 14:23:40.113260 6057 consensus_peers.cc:485] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:23:40.114701 8301 log.cc:824] T 25f9ebd715dc4f96901b60796ea846ad P 3416b273466c495298342b6e892015c2: Log is configured to *not* fsync() on all Append() calls
I20240502 14:23:40.142822 8301 tablet_bootstrap.cc:492] T 25f9ebd715dc4f96901b60796ea846ad P 3416b273466c495298342b6e892015c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:23:40.143226 8301 tablet_bootstrap.cc:492] T 25f9ebd715dc4f96901b60796ea846ad P 3416b273466c495298342b6e892015c2: Bootstrap complete.
I20240502 14:23:40.143559 8301 ts_tablet_manager.cc:1367] T 25f9ebd715dc4f96901b60796ea846ad P 3416b273466c495298342b6e892015c2: Time spent bootstrapping tablet: real 0.038s user 0.035s sys 0.001s
I20240502 14:23:40.147439 8301 raft_consensus.cc:348] T 25f9ebd715dc4f96901b60796ea846ad P 3416b273466c495298342b6e892015c2 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: NON_VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: true } }
I20240502 14:23:40.147761 8301 raft_consensus.cc:729] T 25f9ebd715dc4f96901b60796ea846ad P 3416b273466c495298342b6e892015c2 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 3416b273466c495298342b6e892015c2, State: Initialized, Role: LEARNER
I20240502 14:23:40.148224 8301 consensus_queue.cc:260] T 25f9ebd715dc4f96901b60796ea846ad P 3416b273466c495298342b6e892015c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: NON_VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: true } }
I20240502 14:23:40.149439 8301 ts_tablet_manager.cc:1398] T 25f9ebd715dc4f96901b60796ea846ad P 3416b273466c495298342b6e892015c2: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20240502 14:23:40.150262 6144 tablet_copy_service.cc:342] P 333acff728274715a9816c286921ddd0: Request end of tablet copy session 3416b273466c495298342b6e892015c2-25f9ebd715dc4f96901b60796ea846ad received from {username='slave'} at 127.5.107.69:60319
I20240502 14:23:40.150368 6144 tablet_copy_service.cc:433] P 333acff728274715a9816c286921ddd0: ending tablet copy session 3416b273466c495298342b6e892015c2-25f9ebd715dc4f96901b60796ea846ad on tablet 25f9ebd715dc4f96901b60796ea846ad with peer 3416b273466c495298342b6e892015c2
W20240502 14:23:40.189960 5927 consensus_peers.cc:485] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:23:40.251413 5797 consensus_peers.cc:485] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:23:40.258512 7242 raft_consensus.cc:1203] T 25f9ebd715dc4f96901b60796ea846ad P 3416b273466c495298342b6e892015c2 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.5->[2.6-2.6] Dedup: 2.6->[]
W20240502 14:23:40.413831 6057 consensus_peers.cc:485] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:23:40.458899 6057 consensus_peers.cc:485] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:23:40.733541 5797 consensus_peers.cc:485] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:23:40.787837 8317 raft_consensus.cc:1050] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0: attempting to promote NON_VOTER 3416b273466c495298342b6e892015c2 to VOTER
I20240502 14:23:40.788798 8317 consensus_queue.cc:237] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } }
I20240502 14:23:40.791111 7242 raft_consensus.cc:1261] T 25f9ebd715dc4f96901b60796ea846ad P 3416b273466c495298342b6e892015c2 [term 2 LEARNER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:23:40.791182 5864 raft_consensus.cc:1261] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:23:40.791709 8358 consensus_queue.cc:1035] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:23:40.791915 8317 consensus_queue.cc:1035] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:23:40.792358 6057 consensus_peers.cc:485] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:23:40.795423 8358 raft_consensus.cc:2932] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 3416b273466c495298342b6e892015c2 (127.5.107.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } }
I20240502 14:23:40.796074 7242 raft_consensus.cc:2932] T 25f9ebd715dc4f96901b60796ea846ad P 3416b273466c495298342b6e892015c2 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 3416b273466c495298342b6e892015c2 (127.5.107.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } }
I20240502 14:23:40.805481 5586 catalog_manager.cc:5484] T 25f9ebd715dc4f96901b60796ea846ad P 3416b273466c495298342b6e892015c2 reported cstate change: config changed from index 6 to 7, 3416b273466c495298342b6e892015c2 (127.5.107.69) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } }
I20240502 14:23:40.805766 5864 raft_consensus.cc:2932] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 3416b273466c495298342b6e892015c2 (127.5.107.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } }
I20240502 14:23:40.817659 6123 consensus_queue.cc:237] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } }
I20240502 14:23:40.819283 7242 raft_consensus.cc:1261] T 25f9ebd715dc4f96901b60796ea846ad P 3416b273466c495298342b6e892015c2 [term 2 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:23:40.819414 5864 raft_consensus.cc:1261] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:23:40.819839 8369 consensus_queue.cc:1035] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:23:40.820426 8369 consensus_queue.cc:1035] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:23:40.821684 8358 raft_consensus.cc:2932] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } }
I20240502 14:23:40.822333 5864 raft_consensus.cc:2932] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } }
I20240502 14:23:40.822868 7242 raft_consensus.cc:2932] T 25f9ebd715dc4f96901b60796ea846ad P 3416b273466c495298342b6e892015c2 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } }
I20240502 14:23:40.824766 5578 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 25f9ebd715dc4f96901b60796ea846ad with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:23:40.826620 5591 catalog_manager.cc:5484] T 25f9ebd715dc4f96901b60796ea846ad P 3416b273466c495298342b6e892015c2 reported cstate change: config changed from index 7 to 8, VOTER 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65) evicted. New cstate: current_term: 2 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } }
W20240502 14:23:41.017129 5576 catalog_manager.cc:4573] TS 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 25f9ebd715dc4f96901b60796ea846ad: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
W20240502 14:23:41.043437 5927 consensus_peers.cc:485] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:23:41.141593 6123 consensus_queue.cc:237] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:41.143589 5993 raft_consensus.cc:1261] T 9fdda89a7743466a8aeceb543124fd79 P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:23:41.144096 8373 consensus_queue.cc:1035] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:23:41.146642 8369 raft_consensus.cc:2932] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } }
I20240502 14:23:41.147162 5993 raft_consensus.cc:2932] T 9fdda89a7743466a8aeceb543124fd79 P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } }
I20240502 14:23:41.151638 5589 catalog_manager.cc:5484] T 9fdda89a7743466a8aeceb543124fd79 P 181b466de0084fcbace0ca327fb1cbef reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } }
W20240502 14:23:41.162578 6057 consensus_peers.cc:485] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:23:41.167531 6123 consensus_queue.cc:237] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: NON_VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: true } }
I20240502 14:23:41.171022 5993 raft_consensus.cc:1261] T 9fdda89a7743466a8aeceb543124fd79 P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:23:41.171773 8373 consensus_queue.cc:1035] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:23:41.174091 8373 raft_consensus.cc:2932] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: NON_VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: true } } }
I20240502 14:23:41.174857 5993 raft_consensus.cc:2932] T 9fdda89a7743466a8aeceb543124fd79 P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: NON_VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: true } } }
I20240502 14:23:41.178216 5578 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9fdda89a7743466a8aeceb543124fd79 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240502 14:23:41.179279 6057 consensus_peers.cc:485] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:23:41.179527 5589 catalog_manager.cc:5484] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 reported cstate change: config changed from index 4 to 5, NON_VOTER 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) added. New cstate: current_term: 1 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: NON_VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:23:41.180370 6060 consensus_peers.cc:485] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 -> Peer 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71:42329): Couldn't send request to peer 681fd3c49e914094bf28a7df155cffc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9fdda89a7743466a8aeceb543124fd79. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:23:41.514231 6057 consensus_peers.cc:485] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:23:41.789783 8411 ts_tablet_manager.cc:902] T 9fdda89a7743466a8aeceb543124fd79 P 681fd3c49e914094bf28a7df155cffc9: Initiating tablet copy from peer 333acff728274715a9816c286921ddd0 (127.5.107.68:45913)
I20240502 14:23:41.791265 8411 tablet_copy_client.cc:308] T 9fdda89a7743466a8aeceb543124fd79 P 681fd3c49e914094bf28a7df155cffc9: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.68:45913
I20240502 14:23:41.796664 6144 tablet_copy_service.cc:140] P 333acff728274715a9816c286921ddd0: Received BeginTabletCopySession request for tablet 9fdda89a7743466a8aeceb543124fd79 from peer 681fd3c49e914094bf28a7df155cffc9 ({username='slave'} at 127.5.107.71:48765)
I20240502 14:23:41.796797 6144 tablet_copy_service.cc:161] P 333acff728274715a9816c286921ddd0: Beginning new tablet copy session on tablet 9fdda89a7743466a8aeceb543124fd79 from peer 681fd3c49e914094bf28a7df155cffc9 at {username='slave'} at 127.5.107.71:48765: session id = 681fd3c49e914094bf28a7df155cffc9-9fdda89a7743466a8aeceb543124fd79
I20240502 14:23:41.798715 6144 tablet_copy_source_session.cc:205] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:23:41.803546 8411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fdda89a7743466a8aeceb543124fd79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:41.814680 8411 tablet_copy_client.cc:783] T 9fdda89a7743466a8aeceb543124fd79 P 681fd3c49e914094bf28a7df155cffc9: tablet copy: Starting download of 0 data blocks...
I20240502 14:23:41.814965 8411 tablet_copy_client.cc:647] T 9fdda89a7743466a8aeceb543124fd79 P 681fd3c49e914094bf28a7df155cffc9: tablet copy: Starting download of 1 WAL segments...
I20240502 14:23:41.816920 8415 tablet_copy_client.cc:1002] T 9fdda89a7743466a8aeceb543124fd79 P 681fd3c49e914094bf28a7df155cffc9: tablet copy: Injecting latency into file download: 1500
W20240502 14:23:41.917637 5927 consensus_peers.cc:485] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:23:42.069339 5576 catalog_manager.cc:4573] TS 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7d944a63fdf24ecd8fadbad3b620af0a: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
W20240502 14:23:42.307818 6057 consensus_peers.cc:485] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:23:42.539233 6057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 14:23:42.639797 5797 consensus_peers.cc:485] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:23:42.677945 5927 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:23:42.678802 5927 consensus_peers.cc:485] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:23:42.838891 5797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:23:43.112607 6057 consensus_peers.cc:485] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:23:43.269997 5797 consensus_peers.cc:485] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:23:43.317449 8411 tablet_copy_client.cc:523] T 9fdda89a7743466a8aeceb543124fd79 P 681fd3c49e914094bf28a7df155cffc9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:23:43.320936 8411 tablet_bootstrap.cc:492] T 9fdda89a7743466a8aeceb543124fd79 P 681fd3c49e914094bf28a7df155cffc9: Bootstrap starting.
I20240502 14:23:43.329473 8411 log.cc:824] T 9fdda89a7743466a8aeceb543124fd79 P 681fd3c49e914094bf28a7df155cffc9: Log is configured to *not* fsync() on all Append() calls
I20240502 14:23:43.357107 8411 tablet_bootstrap.cc:492] T 9fdda89a7743466a8aeceb543124fd79 P 681fd3c49e914094bf28a7df155cffc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:23:43.357571 8411 tablet_bootstrap.cc:492] T 9fdda89a7743466a8aeceb543124fd79 P 681fd3c49e914094bf28a7df155cffc9: Bootstrap complete.
I20240502 14:23:43.357957 8411 ts_tablet_manager.cc:1367] T 9fdda89a7743466a8aeceb543124fd79 P 681fd3c49e914094bf28a7df155cffc9: Time spent bootstrapping tablet: real 0.037s user 0.035s sys 0.004s
I20240502 14:23:43.362123 8411 raft_consensus.cc:348] T 9fdda89a7743466a8aeceb543124fd79 P 681fd3c49e914094bf28a7df155cffc9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: NON_VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: true } }
I20240502 14:23:43.362455 8411 raft_consensus.cc:729] T 9fdda89a7743466a8aeceb543124fd79 P 681fd3c49e914094bf28a7df155cffc9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 681fd3c49e914094bf28a7df155cffc9, State: Initialized, Role: LEARNER
I20240502 14:23:43.362871 8411 consensus_queue.cc:260] T 9fdda89a7743466a8aeceb543124fd79 P 681fd3c49e914094bf28a7df155cffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: NON_VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: true } }
I20240502 14:23:43.364056 8411 ts_tablet_manager.cc:1398] T 9fdda89a7743466a8aeceb543124fd79 P 681fd3c49e914094bf28a7df155cffc9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 14:23:43.364924 6144 tablet_copy_service.cc:342] P 333acff728274715a9816c286921ddd0: Request end of tablet copy session 681fd3c49e914094bf28a7df155cffc9-9fdda89a7743466a8aeceb543124fd79 received from {username='slave'} at 127.5.107.71:48765
I20240502 14:23:43.365056 6144 tablet_copy_service.cc:433] P 333acff728274715a9816c286921ddd0: ending tablet copy session 681fd3c49e914094bf28a7df155cffc9-9fdda89a7743466a8aeceb543124fd79 on tablet 9fdda89a7743466a8aeceb543124fd79 with peer 681fd3c49e914094bf28a7df155cffc9
W20240502 14:23:43.593874 5927 consensus_peers.cc:485] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:23:43.646749 6057 consensus_peers.cc:485] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:23:43.718341 7504 raft_consensus.cc:1203] T 9fdda89a7743466a8aeceb543124fd79 P 681fd3c49e914094bf28a7df155cffc9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240502 14:23:44.110513 6057 consensus_peers.cc:485] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:23:44.197144 5927 consensus_peers.cc:485] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:23:44.327212 8486 raft_consensus.cc:1050] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0: attempting to promote NON_VOTER 681fd3c49e914094bf28a7df155cffc9 to VOTER
I20240502 14:23:44.327853 8486 consensus_queue.cc:237] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } }
I20240502 14:23:44.330222 5993 raft_consensus.cc:1261] T 9fdda89a7743466a8aeceb543124fd79 P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:23:44.330833 7504 raft_consensus.cc:1261] T 9fdda89a7743466a8aeceb543124fd79 P 681fd3c49e914094bf28a7df155cffc9 [term 1 LEARNER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:23:44.330866 8487 consensus_queue.cc:1035] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:23:44.331331 8487 consensus_queue.cc:1035] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 14:23:44.331605 6057 consensus_peers.cc:485] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:23:44.334969 8430 raft_consensus.cc:2932] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } }
I20240502 14:23:44.335727 7504 raft_consensus.cc:2932] T 9fdda89a7743466a8aeceb543124fd79 P 681fd3c49e914094bf28a7df155cffc9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } }
I20240502 14:23:44.338243 5993 raft_consensus.cc:2932] T 9fdda89a7743466a8aeceb543124fd79 P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } }
I20240502 14:23:44.339416 5586 catalog_manager.cc:5484] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 reported cstate change: config changed from index 5 to 6, 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:23:44.351239 6123 consensus_queue.cc:237] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } }
I20240502 14:23:44.353006 7504 raft_consensus.cc:1261] T 9fdda89a7743466a8aeceb543124fd79 P 681fd3c49e914094bf28a7df155cffc9 [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:23:44.353017 5993 raft_consensus.cc:1261] T 9fdda89a7743466a8aeceb543124fd79 P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:23:44.353554 8489 consensus_queue.cc:1035] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:23:44.353873 8487 consensus_queue.cc:1035] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:23:44.356122 8486 raft_consensus.cc:2932] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } }
I20240502 14:23:44.356431 5993 raft_consensus.cc:2932] T 9fdda89a7743466a8aeceb543124fd79 P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } }
I20240502 14:23:44.359659 5578 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9fdda89a7743466a8aeceb543124fd79 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:23:44.360532 5591 catalog_manager.cc:5484] T 9fdda89a7743466a8aeceb543124fd79 P 181b466de0084fcbace0ca327fb1cbef reported cstate change: config changed from index 6 to 7, VOTER 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65) evicted. New cstate: current_term: 1 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } }
I20240502 14:23:44.362012 7504 raft_consensus.cc:2932] T 9fdda89a7743466a8aeceb543124fd79 P 681fd3c49e914094bf28a7df155cffc9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } }
W20240502 14:23:44.366689 5576 catalog_manager.cc:4573] TS 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9fdda89a7743466a8aeceb543124fd79: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
I20240502 14:23:44.585021 5993 consensus_queue.cc:237] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:23:44.587342 6123 raft_consensus.cc:1261] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Refusing update from remote peer 181b466de0084fcbace0ca327fb1cbef: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:23:44.587878 8511 consensus_queue.cc:1035] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Connected to new peer: Peer: permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:23:44.588151 5927 consensus_peers.cc:485] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:23:44.590561 8323 raft_consensus.cc:2932] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } }
I20240502 14:23:44.593792 6123 raft_consensus.cc:2932] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } }
I20240502 14:23:44.598457 5586 catalog_manager.cc:5484] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 reported cstate change: config changed from index -1 to 5. New cstate: current_term: 2 leader_uuid: "181b466de0084fcbace0ca327fb1cbef" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } }
I20240502 14:23:44.603206 5993 consensus_queue.cc:237] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: NON_VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: true } }
I20240502 14:23:44.605228 6123 raft_consensus.cc:1261] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Refusing update from remote peer 181b466de0084fcbace0ca327fb1cbef: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:23:44.606370 8323 consensus_queue.cc:1035] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Connected to new peer: Peer: permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:23:44.608597 8323 raft_consensus.cc:2932] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: NON_VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: true } } }
I20240502 14:23:44.612527 6123 raft_consensus.cc:2932] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: NON_VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: true } } }
I20240502 14:23:44.613615 5591 catalog_manager.cc:5484] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef reported cstate change: config changed from index 5 to 6, NON_VOTER 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) added. New cstate: current_term: 2 leader_uuid: "181b466de0084fcbace0ca327fb1cbef" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: NON_VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:23:44.615224 5927 consensus_peers.cc:485] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:23:44.626521 5578 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet aca9200e5c264c27833d7a4a9122c4d9 with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240502 14:23:44.627940 5930 consensus_peers.cc:485] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef -> Peer 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71:42329): Couldn't send request to peer 681fd3c49e914094bf28a7df155cffc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aca9200e5c264c27833d7a4a9122c4d9. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:23:45.030406 5797 consensus_peers.cc:485] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:23:45.047827 6057 consensus_peers.cc:485] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:23:45.115804 5576 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 16 similar messages]
I20240502 14:23:45.140506 8532 ts_tablet_manager.cc:902] T aca9200e5c264c27833d7a4a9122c4d9 P 681fd3c49e914094bf28a7df155cffc9: Initiating tablet copy from peer 181b466de0084fcbace0ca327fb1cbef (127.5.107.67:41517)
I20240502 14:23:45.141687 8532 tablet_copy_client.cc:308] T aca9200e5c264c27833d7a4a9122c4d9 P 681fd3c49e914094bf28a7df155cffc9: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.67:41517
I20240502 14:23:45.146991 6014 tablet_copy_service.cc:140] P 181b466de0084fcbace0ca327fb1cbef: Received BeginTabletCopySession request for tablet aca9200e5c264c27833d7a4a9122c4d9 from peer 681fd3c49e914094bf28a7df155cffc9 ({username='slave'} at 127.5.107.71:33981)
I20240502 14:23:45.147150 6014 tablet_copy_service.cc:161] P 181b466de0084fcbace0ca327fb1cbef: Beginning new tablet copy session on tablet aca9200e5c264c27833d7a4a9122c4d9 from peer 681fd3c49e914094bf28a7df155cffc9 at {username='slave'} at 127.5.107.71:33981: session id = 681fd3c49e914094bf28a7df155cffc9-aca9200e5c264c27833d7a4a9122c4d9
I20240502 14:23:45.149669 6014 tablet_copy_source_session.cc:205] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:23:45.151218 8532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aca9200e5c264c27833d7a4a9122c4d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:45.155288 8532 tablet_copy_client.cc:783] T aca9200e5c264c27833d7a4a9122c4d9 P 681fd3c49e914094bf28a7df155cffc9: tablet copy: Starting download of 0 data blocks...
I20240502 14:23:45.155488 8532 tablet_copy_client.cc:647] T aca9200e5c264c27833d7a4a9122c4d9 P 681fd3c49e914094bf28a7df155cffc9: tablet copy: Starting download of 1 WAL segments...
I20240502 14:23:45.156946 8536 tablet_copy_client.cc:1002] T aca9200e5c264c27833d7a4a9122c4d9 P 681fd3c49e914094bf28a7df155cffc9: tablet copy: Injecting latency into file download: 1500
W20240502 14:23:45.362324 5927 consensus_peers.cc:485] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:23:45.499089 5576 catalog_manager.cc:4573] TS 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9fdda89a7743466a8aeceb543124fd79: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
W20240502 14:23:45.574165 5797 consensus_peers.cc:485] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:23:45.714771 6057 consensus_peers.cc:485] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:23:46.542593 5576 catalog_manager.cc:4573] TS 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9fdda89a7743466a8aeceb543124fd79: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
I20240502 14:23:46.657296 8532 tablet_copy_client.cc:523] T aca9200e5c264c27833d7a4a9122c4d9 P 681fd3c49e914094bf28a7df155cffc9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:23:46.659868 8532 tablet_bootstrap.cc:492] T aca9200e5c264c27833d7a4a9122c4d9 P 681fd3c49e914094bf28a7df155cffc9: Bootstrap starting.
W20240502 14:23:46.662907 5927 consensus_peers.cc:485] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:23:46.688952 8532 tablet_bootstrap.cc:492] T aca9200e5c264c27833d7a4a9122c4d9 P 681fd3c49e914094bf28a7df155cffc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:23:46.689332 8532 tablet_bootstrap.cc:492] T aca9200e5c264c27833d7a4a9122c4d9 P 681fd3c49e914094bf28a7df155cffc9: Bootstrap complete.
I20240502 14:23:46.689514 8532 ts_tablet_manager.cc:1367] T aca9200e5c264c27833d7a4a9122c4d9 P 681fd3c49e914094bf28a7df155cffc9: Time spent bootstrapping tablet: real 0.030s user 0.023s sys 0.008s
I20240502 14:23:46.690308 8532 raft_consensus.cc:348] T aca9200e5c264c27833d7a4a9122c4d9 P 681fd3c49e914094bf28a7df155cffc9 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: NON_VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: true } }
I20240502 14:23:46.690500 8532 raft_consensus.cc:729] T aca9200e5c264c27833d7a4a9122c4d9 P 681fd3c49e914094bf28a7df155cffc9 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 681fd3c49e914094bf28a7df155cffc9, State: Initialized, Role: LEARNER
I20240502 14:23:46.690637 8532 consensus_queue.cc:260] T aca9200e5c264c27833d7a4a9122c4d9 P 681fd3c49e914094bf28a7df155cffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: NON_VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: true } }
I20240502 14:23:46.691236 8532 ts_tablet_manager.cc:1398] T aca9200e5c264c27833d7a4a9122c4d9 P 681fd3c49e914094bf28a7df155cffc9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:23:46.692027 6014 tablet_copy_service.cc:342] P 181b466de0084fcbace0ca327fb1cbef: Request end of tablet copy session 681fd3c49e914094bf28a7df155cffc9-aca9200e5c264c27833d7a4a9122c4d9 received from {username='slave'} at 127.5.107.71:33981
I20240502 14:23:46.692160 6014 tablet_copy_service.cc:433] P 181b466de0084fcbace0ca327fb1cbef: ending tablet copy session 681fd3c49e914094bf28a7df155cffc9-aca9200e5c264c27833d7a4a9122c4d9 on tablet aca9200e5c264c27833d7a4a9122c4d9 with peer 681fd3c49e914094bf28a7df155cffc9
W20240502 14:23:46.876461 6057 consensus_peers.cc:485] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:23:46.975509 5927 consensus_peers.cc:485] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:23:47.273747 7504 raft_consensus.cc:1203] T aca9200e5c264c27833d7a4a9122c4d9 P 681fd3c49e914094bf28a7df155cffc9 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.5->[2.6-2.6] Dedup: 2.6->[]
W20240502 14:23:47.408701 5797 consensus_peers.cc:485] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:23:47.461002 6057 consensus_peers.cc:485] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:23:47.597033 6057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 34 similar messages]
I20240502 14:23:47.700327 8568 raft_consensus.cc:1050] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef: attempting to promote NON_VOTER 681fd3c49e914094bf28a7df155cffc9 to VOTER
I20240502 14:23:47.701053 8568 consensus_queue.cc:237] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } }
I20240502 14:23:47.703169 7504 raft_consensus.cc:1261] T aca9200e5c264c27833d7a4a9122c4d9 P 681fd3c49e914094bf28a7df155cffc9 [term 2 LEARNER]: Refusing update from remote peer 181b466de0084fcbace0ca327fb1cbef: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:23:47.703189 6123 raft_consensus.cc:1261] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Refusing update from remote peer 181b466de0084fcbace0ca327fb1cbef: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
W20240502 14:23:47.703661 5927 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 14:23:47.703691 8568 consensus_queue.cc:1035] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Connected to new peer: Peer: permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:23:47.704016 8613 consensus_queue.cc:1035] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Connected to new peer: Peer: permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:23:47.704614 5927 consensus_peers.cc:485] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:23:47.707369 8568 raft_consensus.cc:2932] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } }
I20240502 14:23:47.707948 6123 raft_consensus.cc:2932] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } }
I20240502 14:23:47.711424 7504 raft_consensus.cc:2932] T aca9200e5c264c27833d7a4a9122c4d9 P 681fd3c49e914094bf28a7df155cffc9 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } }
I20240502 14:23:47.711891 5586 catalog_manager.cc:5484] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef reported cstate change: config changed from index 6 to 7, 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "181b466de0084fcbace0ca327fb1cbef" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:23:47.718132 5993 consensus_queue.cc:237] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } }
I20240502 14:23:47.719717 7504 raft_consensus.cc:1261] T aca9200e5c264c27833d7a4a9122c4d9 P 681fd3c49e914094bf28a7df155cffc9 [term 2 FOLLOWER]: Refusing update from remote peer 181b466de0084fcbace0ca327fb1cbef: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:23:47.719719 6123 raft_consensus.cc:1261] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Refusing update from remote peer 181b466de0084fcbace0ca327fb1cbef: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:23:47.720185 8611 consensus_queue.cc:1035] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Connected to new peer: Peer: permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:23:47.720393 8613 consensus_queue.cc:1035] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Connected to new peer: Peer: permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:23:47.722334 8613 raft_consensus.cc:2932] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } }
I20240502 14:23:47.722849 7504 raft_consensus.cc:2932] T aca9200e5c264c27833d7a4a9122c4d9 P 681fd3c49e914094bf28a7df155cffc9 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } }
I20240502 14:23:47.723260 6123 raft_consensus.cc:2932] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } }
I20240502 14:23:47.725617 5578 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet aca9200e5c264c27833d7a4a9122c4d9 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:23:47.726673 5589 catalog_manager.cc:5484] T aca9200e5c264c27833d7a4a9122c4d9 P 681fd3c49e914094bf28a7df155cffc9 reported cstate change: config changed from index 7 to 8, VOTER 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65) evicted. New cstate: current_term: 2 leader_uuid: "181b466de0084fcbace0ca327fb1cbef" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } }
W20240502 14:23:47.734840 5576 catalog_manager.cc:4573] TS 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet aca9200e5c264c27833d7a4a9122c4d9: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
W20240502 14:23:47.789047 5927 consensus_peers.cc:485] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:23:47.843022 5797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 14:23:48.062049 5973 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:23:48.073796 7222 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:48.081440 6103 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:23:48.085342 7354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 14:23:48.103104 5797 consensus_peers.cc:485] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:23:48.105242 7484 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:48.117300 7614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 14:23:48.119536 6057 consensus_peers.cc:485] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:23:48.151369 5844 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:23:48.295001 6103 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:23:48.297017 5973 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:23:48.298882 7222 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:48.307984 7354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:48.312314 7484 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:48.317651 5844 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:23:48.327306 7614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:23:48.406474 5992 consensus_queue.cc:237] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } }
I20240502 14:23:48.409385 6123 raft_consensus.cc:1261] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Refusing update from remote peer 181b466de0084fcbace0ca327fb1cbef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:23:48.409508 5864 raft_consensus.cc:1261] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Refusing update from remote peer 181b466de0084fcbace0ca327fb1cbef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:23:48.410041 8708 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Connected to new peer: Peer: permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:23:48.410441 8613 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:23:48.413215 8707 raft_consensus.cc:2932] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } } }
I20240502 14:23:48.414585 6123 raft_consensus.cc:2932] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } } }
I20240502 14:23:48.417951 5864 raft_consensus.cc:2932] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } } }
I20240502 14:23:48.418514 5592 catalog_manager.cc:5484] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef reported cstate change: config changed from index -1 to 4, NON_VOTER 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) added. New cstate: current_term: 1 leader_uuid: "181b466de0084fcbace0ca327fb1cbef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:23:48.418970 5927 consensus_peers.cc:485] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef -> Peer 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72:35323): Couldn't send request to peer 83b0e3c0ba724317832f8505a1ffb66e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5358f204e4324db78adad5af90c17b86. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:23:48.436489 5992 consensus_queue.cc:237] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: NON_VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: true } }
I20240502 14:23:48.438234 7504 raft_consensus.cc:1261] T aca9200e5c264c27833d7a4a9122c4d9 P 681fd3c49e914094bf28a7df155cffc9 [term 2 FOLLOWER]: Refusing update from remote peer 181b466de0084fcbace0ca327fb1cbef: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:23:48.438513 6123 raft_consensus.cc:1261] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Refusing update from remote peer 181b466de0084fcbace0ca327fb1cbef: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:23:48.438859 8613 consensus_queue.cc:1035] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Connected to new peer: Peer: permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:23:48.439138 8707 consensus_queue.cc:1035] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Connected to new peer: Peer: permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:23:48.441360 8613 raft_consensus.cc:2932] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 3416b273466c495298342b6e892015c2 (127.5.107.69) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: NON_VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: true } } }
I20240502 14:23:48.442337 6123 raft_consensus.cc:2932] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 3416b273466c495298342b6e892015c2 (127.5.107.69) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: NON_VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: true } } }
W20240502 14:23:48.444162 5929 consensus_peers.cc:485] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef -> Peer 3416b273466c495298342b6e892015c2 (127.5.107.69:37277): Couldn't send request to peer 3416b273466c495298342b6e892015c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aca9200e5c264c27833d7a4a9122c4d9. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:23:48.445849 7504 raft_consensus.cc:2932] T aca9200e5c264c27833d7a4a9122c4d9 P 681fd3c49e914094bf28a7df155cffc9 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 3416b273466c495298342b6e892015c2 (127.5.107.69) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: NON_VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: true } } }
I20240502 14:23:48.446475 5592 catalog_manager.cc:5484] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 reported cstate change: config changed from index 8 to 9, NON_VOTER 3416b273466c495298342b6e892015c2 (127.5.107.69) added. New cstate: current_term: 2 leader_uuid: "181b466de0084fcbace0ca327fb1cbef" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: NON_VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: true } } }
I20240502 14:23:48.798511 8742 ts_tablet_manager.cc:902] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e: Initiating tablet copy from peer 181b466de0084fcbace0ca327fb1cbef (127.5.107.67:41517)
I20240502 14:23:48.800361 8742 tablet_copy_client.cc:308] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.67:41517
I20240502 14:23:48.806787 6014 tablet_copy_service.cc:140] P 181b466de0084fcbace0ca327fb1cbef: Received BeginTabletCopySession request for tablet 5358f204e4324db78adad5af90c17b86 from peer 83b0e3c0ba724317832f8505a1ffb66e ({username='slave'} at 127.5.107.72:41875)
I20240502 14:23:48.806926 6014 tablet_copy_service.cc:161] P 181b466de0084fcbace0ca327fb1cbef: Beginning new tablet copy session on tablet 5358f204e4324db78adad5af90c17b86 from peer 83b0e3c0ba724317832f8505a1ffb66e at {username='slave'} at 127.5.107.72:41875: session id = 83b0e3c0ba724317832f8505a1ffb66e-5358f204e4324db78adad5af90c17b86
I20240502 14:23:48.808646 6014 tablet_copy_source_session.cc:205] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:23:48.812801 8742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5358f204e4324db78adad5af90c17b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:48.821882 8742 tablet_copy_client.cc:783] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e: tablet copy: Starting download of 0 data blocks...
I20240502 14:23:48.822153 8742 tablet_copy_client.cc:647] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:23:48.823608 8746 tablet_copy_client.cc:1002] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e: tablet copy: Injecting latency into file download: 1500
W20240502 14:23:48.903581 5576 catalog_manager.cc:4573] TS 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet aca9200e5c264c27833d7a4a9122c4d9: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
W20240502 14:23:49.023707 5927 consensus_peers.cc:485] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:23:49.063364 8758 ts_tablet_manager.cc:902] T aca9200e5c264c27833d7a4a9122c4d9 P 3416b273466c495298342b6e892015c2: Initiating tablet copy from peer 181b466de0084fcbace0ca327fb1cbef (127.5.107.67:41517)
I20240502 14:23:49.064536 8758 tablet_copy_client.cc:308] T aca9200e5c264c27833d7a4a9122c4d9 P 3416b273466c495298342b6e892015c2: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.67:41517
I20240502 14:23:49.069713 6014 tablet_copy_service.cc:140] P 181b466de0084fcbace0ca327fb1cbef: Received BeginTabletCopySession request for tablet aca9200e5c264c27833d7a4a9122c4d9 from peer 3416b273466c495298342b6e892015c2 ({username='slave'} at 127.5.107.69:55859)
I20240502 14:23:49.069839 6014 tablet_copy_service.cc:161] P 181b466de0084fcbace0ca327fb1cbef: Beginning new tablet copy session on tablet aca9200e5c264c27833d7a4a9122c4d9 from peer 3416b273466c495298342b6e892015c2 at {username='slave'} at 127.5.107.69:55859: session id = 3416b273466c495298342b6e892015c2-aca9200e5c264c27833d7a4a9122c4d9
I20240502 14:23:49.071534 6014 tablet_copy_source_session.cc:205] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:23:49.072552 8758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aca9200e5c264c27833d7a4a9122c4d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:49.076506 8758 tablet_copy_client.cc:783] T aca9200e5c264c27833d7a4a9122c4d9 P 3416b273466c495298342b6e892015c2: tablet copy: Starting download of 0 data blocks...
I20240502 14:23:49.076751 8758 tablet_copy_client.cc:647] T aca9200e5c264c27833d7a4a9122c4d9 P 3416b273466c495298342b6e892015c2: tablet copy: Starting download of 1 WAL segments...
I20240502 14:23:49.078174 8762 tablet_copy_client.cc:1002] T aca9200e5c264c27833d7a4a9122c4d9 P 3416b273466c495298342b6e892015c2: tablet copy: Injecting latency into file download: 1500
W20240502 14:23:49.456342 5797 consensus_peers.cc:485] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:23:49.604172 6057 consensus_peers.cc:485] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:23:49.966796 5576 catalog_manager.cc:4573] TS 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet aca9200e5c264c27833d7a4a9122c4d9: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
W20240502 14:23:50.282665 6057 consensus_peers.cc:485] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:23:50.298550 5927 consensus_peers.cc:485] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:23:50.324129 8742 tablet_copy_client.cc:523] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:23:50.328238 8742 tablet_bootstrap.cc:492] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e: Bootstrap starting.
I20240502 14:23:50.338115 8742 log.cc:824] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e: Log is configured to *not* fsync() on all Append() calls
I20240502 14:23:50.371210 8742 tablet_bootstrap.cc:492] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:23:50.371656 8742 tablet_bootstrap.cc:492] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e: Bootstrap complete.
I20240502 14:23:50.372011 8742 ts_tablet_manager.cc:1367] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e: Time spent bootstrapping tablet: real 0.044s user 0.035s sys 0.009s
I20240502 14:23:50.376246 8742 raft_consensus.cc:348] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } }
I20240502 14:23:50.376629 8742 raft_consensus.cc:729] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83b0e3c0ba724317832f8505a1ffb66e, State: Initialized, Role: LEARNER
I20240502 14:23:50.377020 8742 consensus_queue.cc:260] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } }
I20240502 14:23:50.378260 8742 ts_tablet_manager.cc:1398] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20240502 14:23:50.379238 6014 tablet_copy_service.cc:342] P 181b466de0084fcbace0ca327fb1cbef: Request end of tablet copy session 83b0e3c0ba724317832f8505a1ffb66e-5358f204e4324db78adad5af90c17b86 received from {username='slave'} at 127.5.107.72:41875
I20240502 14:23:50.379417 6014 tablet_copy_service.cc:433] P 181b466de0084fcbace0ca327fb1cbef: ending tablet copy session 83b0e3c0ba724317832f8505a1ffb66e-5358f204e4324db78adad5af90c17b86 on tablet 5358f204e4324db78adad5af90c17b86 with peer 83b0e3c0ba724317832f8505a1ffb66e
I20240502 14:23:50.578552 8758 tablet_copy_client.cc:523] T aca9200e5c264c27833d7a4a9122c4d9 P 3416b273466c495298342b6e892015c2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:23:50.581130 8758 tablet_bootstrap.cc:492] T aca9200e5c264c27833d7a4a9122c4d9 P 3416b273466c495298342b6e892015c2: Bootstrap starting.
I20240502 14:23:50.611220 8758 tablet_bootstrap.cc:492] T aca9200e5c264c27833d7a4a9122c4d9 P 3416b273466c495298342b6e892015c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:23:50.611593 8758 tablet_bootstrap.cc:492] T aca9200e5c264c27833d7a4a9122c4d9 P 3416b273466c495298342b6e892015c2: Bootstrap complete.
I20240502 14:23:50.611811 8758 ts_tablet_manager.cc:1367] T aca9200e5c264c27833d7a4a9122c4d9 P 3416b273466c495298342b6e892015c2: Time spent bootstrapping tablet: real 0.031s user 0.026s sys 0.005s
I20240502 14:23:50.612579 8758 raft_consensus.cc:348] T aca9200e5c264c27833d7a4a9122c4d9 P 3416b273466c495298342b6e892015c2 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: NON_VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: true } }
I20240502 14:23:50.612766 8758 raft_consensus.cc:729] T aca9200e5c264c27833d7a4a9122c4d9 P 3416b273466c495298342b6e892015c2 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 3416b273466c495298342b6e892015c2, State: Initialized, Role: LEARNER
I20240502 14:23:50.612947 8758 consensus_queue.cc:260] T aca9200e5c264c27833d7a4a9122c4d9 P 3416b273466c495298342b6e892015c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: NON_VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: true } }
I20240502 14:23:50.613665 8758 ts_tablet_manager.cc:1398] T aca9200e5c264c27833d7a4a9122c4d9 P 3416b273466c495298342b6e892015c2: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 14:23:50.614375 6014 tablet_copy_service.cc:342] P 181b466de0084fcbace0ca327fb1cbef: Request end of tablet copy session 3416b273466c495298342b6e892015c2-aca9200e5c264c27833d7a4a9122c4d9 received from {username='slave'} at 127.5.107.69:55859
I20240502 14:23:50.614522 6014 tablet_copy_service.cc:433] P 181b466de0084fcbace0ca327fb1cbef: ending tablet copy session 3416b273466c495298342b6e892015c2-aca9200e5c264c27833d7a4a9122c4d9 on tablet aca9200e5c264c27833d7a4a9122c4d9 with peer 3416b273466c495298342b6e892015c2
I20240502 14:23:50.670058 7242 raft_consensus.cc:1203] T aca9200e5c264c27833d7a4a9122c4d9 P 3416b273466c495298342b6e892015c2 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
W20240502 14:23:50.706100 5797 consensus_peers.cc:485] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:23:50.721879 6057 consensus_peers.cc:485] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:23:50.847378 7634 raft_consensus.cc:1203] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240502 14:23:50.960979 5576 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 13 similar messages]
I20240502 14:23:51.106614 8875 raft_consensus.cc:1050] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef: attempting to promote NON_VOTER 3416b273466c495298342b6e892015c2 to VOTER
I20240502 14:23:51.107475 8875 consensus_queue.cc:237] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } }
I20240502 14:23:51.109925 7504 raft_consensus.cc:1261] T aca9200e5c264c27833d7a4a9122c4d9 P 681fd3c49e914094bf28a7df155cffc9 [term 2 FOLLOWER]: Refusing update from remote peer 181b466de0084fcbace0ca327fb1cbef: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:23:51.109953 7242 raft_consensus.cc:1261] T aca9200e5c264c27833d7a4a9122c4d9 P 3416b273466c495298342b6e892015c2 [term 2 LEARNER]: Refusing update from remote peer 181b466de0084fcbace0ca327fb1cbef: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:23:51.110116 6123 raft_consensus.cc:1261] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Refusing update from remote peer 181b466de0084fcbace0ca327fb1cbef: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:23:51.110512 8875 consensus_queue.cc:1035] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Connected to new peer: Peer: permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:23:51.110857 8877 consensus_queue.cc:1035] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Connected to new peer: Peer: permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:23:51.111203 8874 consensus_queue.cc:1035] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Connected to new peer: Peer: permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:23:51.113920 8877 raft_consensus.cc:2932] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 3416b273466c495298342b6e892015c2 (127.5.107.69) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } }
I20240502 14:23:51.114593 6123 raft_consensus.cc:2932] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 3416b273466c495298342b6e892015c2 (127.5.107.69) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } }
I20240502 14:23:51.115259 7242 raft_consensus.cc:2932] T aca9200e5c264c27833d7a4a9122c4d9 P 3416b273466c495298342b6e892015c2 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 3416b273466c495298342b6e892015c2 (127.5.107.69) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } }
I20240502 14:23:51.117969 7504 raft_consensus.cc:2932] T aca9200e5c264c27833d7a4a9122c4d9 P 681fd3c49e914094bf28a7df155cffc9 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 3416b273466c495298342b6e892015c2 (127.5.107.69) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } }
I20240502 14:23:51.118986 5583 catalog_manager.cc:5484] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef reported cstate change: config changed from index 9 to 10, 3416b273466c495298342b6e892015c2 (127.5.107.69) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "181b466de0084fcbace0ca327fb1cbef" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:23:51.126168 5992 consensus_queue.cc:237] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } }
I20240502 14:23:51.127934 7504 raft_consensus.cc:1261] T aca9200e5c264c27833d7a4a9122c4d9 P 681fd3c49e914094bf28a7df155cffc9 [term 2 FOLLOWER]: Refusing update from remote peer 181b466de0084fcbace0ca327fb1cbef: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 14:23:51.127969 7242 raft_consensus.cc:1261] T aca9200e5c264c27833d7a4a9122c4d9 P 3416b273466c495298342b6e892015c2 [term 2 FOLLOWER]: Refusing update from remote peer 181b466de0084fcbace0ca327fb1cbef: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 14:23:51.128715 8874 consensus_queue.cc:1035] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Connected to new peer: Peer: permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:23:51.129033 8877 consensus_queue.cc:1035] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Connected to new peer: Peer: permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:23:51.131091 8875 raft_consensus.cc:2932] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 333acff728274715a9816c286921ddd0 (127.5.107.68) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } }
I20240502 14:23:51.131631 7242 raft_consensus.cc:2932] T aca9200e5c264c27833d7a4a9122c4d9 P 3416b273466c495298342b6e892015c2 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 333acff728274715a9816c286921ddd0 (127.5.107.68) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } }
I20240502 14:23:51.134050 5578 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet aca9200e5c264c27833d7a4a9122c4d9 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:23:51.134661 7504 raft_consensus.cc:2932] T aca9200e5c264c27833d7a4a9122c4d9 P 681fd3c49e914094bf28a7df155cffc9 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 333acff728274715a9816c286921ddd0 (127.5.107.68) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } }
I20240502 14:23:51.135103 5591 catalog_manager.cc:5484] T aca9200e5c264c27833d7a4a9122c4d9 P 181b466de0084fcbace0ca327fb1cbef reported cstate change: config changed from index 10 to 11, VOTER 333acff728274715a9816c286921ddd0 (127.5.107.68) evicted. New cstate: current_term: 2 leader_uuid: "181b466de0084fcbace0ca327fb1cbef" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:23:51.138630 6103 tablet_service.cc:1514] Processing DeleteTablet for tablet aca9200e5c264c27833d7a4a9122c4d9 with delete_type TABLET_DATA_TOMBSTONED (TS 333acff728274715a9816c286921ddd0 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:44714
I20240502 14:23:51.141669 8886 tablet_replica.cc:330] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0: stopping tablet replica
I20240502 14:23:51.142110 8886 raft_consensus.cc:2229] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:23:51.142529 8886 raft_consensus.cc:2258] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:23:51.144142 8886 ts_tablet_manager.cc:1875] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:23:51.148816 8886 ts_tablet_manager.cc:1888] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240502 14:23:51.148954 8886 log.cc:1196] T aca9200e5c264c27833d7a4a9122c4d9 P 333acff728274715a9816c286921ddd0: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal/wals/aca9200e5c264c27833d7a4a9122c4d9
I20240502 14:23:51.149916 5578 catalog_manager.cc:4830] TS 333acff728274715a9816c286921ddd0 (127.5.107.68:45913): tablet aca9200e5c264c27833d7a4a9122c4d9 (table rebalance_test_table_0 [id=7a7408907bfc4fd2bd39cf4e8b4c97fb]) successfully deleted
I20240502 14:23:51.325505 6123 consensus_queue.cc:237] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } }
W20240502 14:23:51.328006 6059 consensus_peers.cc:485] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 -> Peer 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70:40027): Couldn't send request to peer 806206bb049040bf8db48a7d8d3840c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9fdda89a7743466a8aeceb543124fd79. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:23:51.328089 7504 raft_consensus.cc:1261] T 9fdda89a7743466a8aeceb543124fd79 P 681fd3c49e914094bf28a7df155cffc9 [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:23:51.328697 8741 consensus_queue.cc:1035] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:23:51.328791 5992 raft_consensus.cc:1261] T 9fdda89a7743466a8aeceb543124fd79 P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:23:51.329622 8741 consensus_queue.cc:1035] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:23:51.332258 8741 raft_consensus.cc:2932] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } } }
I20240502 14:23:51.332638 7504 raft_consensus.cc:2932] T 9fdda89a7743466a8aeceb543124fd79 P 681fd3c49e914094bf28a7df155cffc9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } } }
I20240502 14:23:51.336524 5992 raft_consensus.cc:2932] T 9fdda89a7743466a8aeceb543124fd79 P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } } }
I20240502 14:23:51.336611 5586 catalog_manager.cc:5484] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 reported cstate change: config changed from index 7 to 8, NON_VOTER 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) added. New cstate: current_term: 1 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:23:51.430464 8875 raft_consensus.cc:1050] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef: attempting to promote NON_VOTER 83b0e3c0ba724317832f8505a1ffb66e to VOTER
I20240502 14:23:51.431314 8875 consensus_queue.cc:237] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } }
I20240502 14:23:51.433111 5863 raft_consensus.cc:1261] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Refusing update from remote peer 181b466de0084fcbace0ca327fb1cbef: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:23:51.433266 6123 raft_consensus.cc:1261] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Refusing update from remote peer 181b466de0084fcbace0ca327fb1cbef: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:23:51.433315 7634 raft_consensus.cc:1261] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e [term 1 LEARNER]: Refusing update from remote peer 181b466de0084fcbace0ca327fb1cbef: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:23:51.433650 8874 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:23:51.433943 8875 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Connected to new peer: Peer: permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:23:51.434165 8775 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:23:51.437203 8775 raft_consensus.cc:2932] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:23:51.437798 6123 raft_consensus.cc:2932] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:23:51.440722 5863 raft_consensus.cc:2932] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:23:51.443343 5591 catalog_manager.cc:5484] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 reported cstate change: config changed from index 4 to 5, 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "181b466de0084fcbace0ca327fb1cbef" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:23:51.446043 7634 raft_consensus.cc:2932] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
W20240502 14:23:51.566601 5927 consensus_peers.cc:485] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:23:51.583555 5992 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5358f204e4324db78adad5af90c17b86"
dest_uuid: "181b466de0084fcbace0ca327fb1cbef"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60932
I20240502 14:23:51.583832 5992 raft_consensus.cc:595] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [term 1 LEADER]: Received request to transfer leadership
I20240502 14:23:51.802276 8946 ts_tablet_manager.cc:902] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1: Initiating tablet copy from peer 333acff728274715a9816c286921ddd0 (127.5.107.68:45913)
I20240502 14:23:51.803951 8946 tablet_copy_client.cc:308] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.68:45913
I20240502 14:23:51.804527 6144 tablet_copy_service.cc:140] P 333acff728274715a9816c286921ddd0: Received BeginTabletCopySession request for tablet 9fdda89a7743466a8aeceb543124fd79 from peer 806206bb049040bf8db48a7d8d3840c1 ({username='slave'} at 127.5.107.70:36379)
I20240502 14:23:51.804688 6144 tablet_copy_service.cc:161] P 333acff728274715a9816c286921ddd0: Beginning new tablet copy session on tablet 9fdda89a7743466a8aeceb543124fd79 from peer 806206bb049040bf8db48a7d8d3840c1 at {username='slave'} at 127.5.107.70:36379: session id = 806206bb049040bf8db48a7d8d3840c1-9fdda89a7743466a8aeceb543124fd79
I20240502 14:23:51.806905 6144 tablet_copy_source_session.cc:205] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:23:51.808058 8946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fdda89a7743466a8aeceb543124fd79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:51.814242 8946 tablet_copy_client.cc:783] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1: tablet copy: Starting download of 0 data blocks...
I20240502 14:23:51.814492 8946 tablet_copy_client.cc:647] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1: tablet copy: Starting download of 1 WAL segments...
W20240502 14:23:51.816036 5797 consensus_peers.cc:485] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:23:51.816143 8953 tablet_copy_client.cc:1002] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1: tablet copy: Injecting latency into file download: 1500
I20240502 14:23:51.823366 5992 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5358f204e4324db78adad5af90c17b86"
dest_uuid: "181b466de0084fcbace0ca327fb1cbef"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60948
I20240502 14:23:51.823530 5992 raft_consensus.cc:595] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [term 1 LEADER]: Received request to transfer leadership
I20240502 14:23:51.879359 8922 raft_consensus.cc:979] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef: : Instructing follower 83e62d4e5f8a4316bc1bcb8c18458081 to start an election
I20240502 14:23:51.879534 8922 raft_consensus.cc:1067] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [term 1 LEADER]: Signalling peer 83e62d4e5f8a4316bc1bcb8c18458081 to start an election
I20240502 14:23:51.880316 5863 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5358f204e4324db78adad5af90c17b86"
dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081"
from {username='slave'} at 127.5.107.67:55771
I20240502 14:23:51.880553 5863 raft_consensus.cc:482] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:23:51.880656 5863 raft_consensus.cc:3037] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:23:51.882653 5863 raft_consensus.cc:504] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } }
I20240502 14:23:51.883610 5863 leader_election.cc:288] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [CANDIDATE]: Term 2 election: Requested vote from peers 333acff728274715a9816c286921ddd0 (127.5.107.68:45913), 181b466de0084fcbace0ca327fb1cbef (127.5.107.67:41517), 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72:35323)
I20240502 14:23:51.883826 6123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5358f204e4324db78adad5af90c17b86" candidate_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "333acff728274715a9816c286921ddd0"
I20240502 14:23:51.883893 5992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5358f204e4324db78adad5af90c17b86" candidate_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "181b466de0084fcbace0ca327fb1cbef"
I20240502 14:23:51.884050 6123 raft_consensus.cc:3037] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:23:51.884073 5992 raft_consensus.cc:3032] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:23:51.884155 5992 raft_consensus.cc:729] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 181b466de0084fcbace0ca327fb1cbef, State: Running, Role: LEADER
I20240502 14:23:51.884428 5992 consensus_queue.cc:260] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } }
I20240502 14:23:51.884868 5992 raft_consensus.cc:3037] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:23:51.886451 6123 raft_consensus.cc:2454] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e62d4e5f8a4316bc1bcb8c18458081 in term 2.
I20240502 14:23:51.887213 5992 raft_consensus.cc:2454] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e62d4e5f8a4316bc1bcb8c18458081 in term 2.
I20240502 14:23:51.887629 5799 leader_election.cc:302] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 181b466de0084fcbace0ca327fb1cbef, 333acff728274715a9816c286921ddd0, 83e62d4e5f8a4316bc1bcb8c18458081; no voters:
I20240502 14:23:51.887936 8543 raft_consensus.cc:2781] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:23:51.888370 8543 raft_consensus.cc:686] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 LEADER]: Becoming Leader. State: Replica: 83e62d4e5f8a4316bc1bcb8c18458081, State: Running, Role: LEADER
I20240502 14:23:51.888746 8543 consensus_queue.cc:237] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } }
I20240502 14:23:51.889874 7634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5358f204e4324db78adad5af90c17b86" candidate_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "83b0e3c0ba724317832f8505a1ffb66e"
I20240502 14:23:51.890210 7634 raft_consensus.cc:3037] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:23:51.891674 5591 catalog_manager.cc:5484] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 reported cstate change: term changed from 1 to 2, leader changed from 181b466de0084fcbace0ca327fb1cbef (127.5.107.67) to 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66). New cstate: current_term: 2 leader_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:23:51.892700 7634 raft_consensus.cc:2454] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e62d4e5f8a4316bc1bcb8c18458081 in term 2.
W20240502 14:23:52.041939 5576 catalog_manager.cc:4573] TS 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet aca9200e5c264c27833d7a4a9122c4d9: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
I20240502 14:23:52.390394 7634 raft_consensus.cc:1261] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e [term 2 FOLLOWER]: Refusing update from remote peer 83e62d4e5f8a4316bc1bcb8c18458081: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:23:52.391321 8543 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:23:52.408074 6123 raft_consensus.cc:1261] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Refusing update from remote peer 83e62d4e5f8a4316bc1bcb8c18458081: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:23:52.408572 8543 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:23:52.414485 5992 raft_consensus.cc:1261] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [term 2 FOLLOWER]: Refusing update from remote peer 83e62d4e5f8a4316bc1bcb8c18458081: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:23:52.415694 8543 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:23:52.417475 5863 consensus_queue.cc:237] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } }
I20240502 14:23:52.419276 7634 raft_consensus.cc:1261] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e [term 2 FOLLOWER]: Refusing update from remote peer 83e62d4e5f8a4316bc1bcb8c18458081: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:23:52.419358 6122 raft_consensus.cc:1261] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Refusing update from remote peer 83e62d4e5f8a4316bc1bcb8c18458081: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:23:52.419843 8989 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:23:52.420184 8543 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:23:52.422363 8989 raft_consensus.cc:2932] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 181b466de0084fcbace0ca327fb1cbef (127.5.107.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:23:52.423035 7634 raft_consensus.cc:2932] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 181b466de0084fcbace0ca327fb1cbef (127.5.107.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:23:52.423086 6122 raft_consensus.cc:2932] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 181b466de0084fcbace0ca327fb1cbef (127.5.107.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:23:52.425936 5577 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5358f204e4324db78adad5af90c17b86 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:23:52.427377 5586 catalog_manager.cc:5484] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e reported cstate change: config changed from index 5 to 7, VOTER 181b466de0084fcbace0ca327fb1cbef (127.5.107.67) evicted. New cstate: current_term: 2 leader_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
W20240502 14:23:52.430364 6057 consensus_peers.cc:485] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:23:52.431787 5973 tablet_service.cc:1514] Processing DeleteTablet for tablet 5358f204e4324db78adad5af90c17b86 with delete_type TABLET_DATA_TOMBSTONED (TS 181b466de0084fcbace0ca327fb1cbef not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:36800
I20240502 14:23:52.432504 8998 tablet_replica.cc:330] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef: stopping tablet replica
I20240502 14:23:52.432960 8998 raft_consensus.cc:2229] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:23:52.433246 8998 raft_consensus.cc:2258] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:23:52.434706 8998 ts_tablet_manager.cc:1875] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:23:52.439155 8998 ts_tablet_manager.cc:1888] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 14:23:52.439302 8998 log.cc:1196] T 5358f204e4324db78adad5af90c17b86 P 181b466de0084fcbace0ca327fb1cbef: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal/wals/5358f204e4324db78adad5af90c17b86
I20240502 14:23:52.440196 5578 catalog_manager.cc:4830] TS 181b466de0084fcbace0ca327fb1cbef (127.5.107.67:41517): tablet 5358f204e4324db78adad5af90c17b86 (table rebalance_test_table_3 [id=5a9100430c684a96b7fd7563ec356139]) successfully deleted
I20240502 14:23:52.486685 6122 consensus_queue.cc:237] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } }
I20240502 14:23:52.488754 5863 raft_consensus.cc:1261] T 08d82302438544e4925a4f511f13fb7e P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:23:52.488987 5992 raft_consensus.cc:1261] T 08d82302438544e4925a4f511f13fb7e P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:23:52.489346 8985 consensus_queue.cc:1035] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:23:52.490128 8985 consensus_queue.cc:1035] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:23:52.493134 8904 raft_consensus.cc:2932] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } } }
I20240502 14:23:52.493594 5992 raft_consensus.cc:2932] T 08d82302438544e4925a4f511f13fb7e P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } } }
I20240502 14:23:52.493800 5863 raft_consensus.cc:2932] T 08d82302438544e4925a4f511f13fb7e P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } } }
I20240502 14:23:52.497753 5591 catalog_manager.cc:5484] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 reported cstate change: config changed from index -1 to 4, NON_VOTER 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) added. New cstate: current_term: 1 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:23:52.498111 6057 consensus_peers.cc:485] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 -> Peer 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72:35323): Couldn't send request to peer 83b0e3c0ba724317832f8505a1ffb66e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08d82302438544e4925a4f511f13fb7e. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:23:52.510953 5863 consensus_queue.cc:237] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: NON_VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: true } }
I20240502 14:23:52.513223 6122 raft_consensus.cc:1261] T 81cfc8c8608b4cb0b7c303a910791d4b P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Refusing update from remote peer 83e62d4e5f8a4316bc1bcb8c18458081: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:23:52.513273 5992 raft_consensus.cc:1261] T 81cfc8c8608b4cb0b7c303a910791d4b P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Refusing update from remote peer 83e62d4e5f8a4316bc1bcb8c18458081: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:23:52.513757 8988 consensus_queue.cc:1035] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:23:52.514587 8543 consensus_queue.cc:1035] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:23:52.516218 8988 raft_consensus.cc:2932] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3416b273466c495298342b6e892015c2 (127.5.107.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: NON_VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: true } } }
W20240502 14:23:52.519351 5799 consensus_peers.cc:485] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 3416b273466c495298342b6e892015c2 (127.5.107.69:37277): Couldn't send request to peer 3416b273466c495298342b6e892015c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81cfc8c8608b4cb0b7c303a910791d4b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:23:52.519886 5992 raft_consensus.cc:2932] T 81cfc8c8608b4cb0b7c303a910791d4b P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3416b273466c495298342b6e892015c2 (127.5.107.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: NON_VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: true } } }
I20240502 14:23:52.521095 6122 raft_consensus.cc:2932] T 81cfc8c8608b4cb0b7c303a910791d4b P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3416b273466c495298342b6e892015c2 (127.5.107.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: NON_VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: true } } }
I20240502 14:23:52.524467 5592 catalog_manager.cc:5484] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 reported cstate change: config changed from index -1 to 4, NON_VOTER 3416b273466c495298342b6e892015c2 (127.5.107.69) added. New cstate: current_term: 1 leader_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: NON_VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:23:52.618927 6057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 26 similar messages]
I20240502 14:23:52.927525 9059 ts_tablet_manager.cc:902] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2: Initiating tablet copy from peer 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015)
I20240502 14:23:52.928679 9059 tablet_copy_client.cc:308] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:34015
I20240502 14:23:52.934751 5884 tablet_copy_service.cc:140] P 83e62d4e5f8a4316bc1bcb8c18458081: Received BeginTabletCopySession request for tablet 81cfc8c8608b4cb0b7c303a910791d4b from peer 3416b273466c495298342b6e892015c2 ({username='slave'} at 127.5.107.69:48521)
I20240502 14:23:52.934937 5884 tablet_copy_service.cc:161] P 83e62d4e5f8a4316bc1bcb8c18458081: Beginning new tablet copy session on tablet 81cfc8c8608b4cb0b7c303a910791d4b from peer 3416b273466c495298342b6e892015c2 at {username='slave'} at 127.5.107.69:48521: session id = 3416b273466c495298342b6e892015c2-81cfc8c8608b4cb0b7c303a910791d4b
I20240502 14:23:52.937541 5884 tablet_copy_source_session.cc:205] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:23:52.939131 9059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81cfc8c8608b4cb0b7c303a910791d4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:52.943338 9059 tablet_copy_client.cc:783] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2: tablet copy: Starting download of 0 data blocks...
I20240502 14:23:52.943589 9059 tablet_copy_client.cc:647] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2: tablet copy: Starting download of 1 WAL segments...
I20240502 14:23:52.945220 9063 tablet_copy_client.cc:1002] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2: tablet copy: Injecting latency into file download: 1500
W20240502 14:23:52.948987 5797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:23:52.998577 6057 consensus_peers.cc:485] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:23:53.015517 9082 ts_tablet_manager.cc:902] T 08d82302438544e4925a4f511f13fb7e P 83b0e3c0ba724317832f8505a1ffb66e: Initiating tablet copy from peer 333acff728274715a9816c286921ddd0 (127.5.107.68:45913)
I20240502 14:23:53.016499 9082 tablet_copy_client.cc:308] T 08d82302438544e4925a4f511f13fb7e P 83b0e3c0ba724317832f8505a1ffb66e: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.68:45913
I20240502 14:23:53.021728 6144 tablet_copy_service.cc:140] P 333acff728274715a9816c286921ddd0: Received BeginTabletCopySession request for tablet 08d82302438544e4925a4f511f13fb7e from peer 83b0e3c0ba724317832f8505a1ffb66e ({username='slave'} at 127.5.107.72:46611)
I20240502 14:23:53.021857 6144 tablet_copy_service.cc:161] P 333acff728274715a9816c286921ddd0: Beginning new tablet copy session on tablet 08d82302438544e4925a4f511f13fb7e from peer 83b0e3c0ba724317832f8505a1ffb66e at {username='slave'} at 127.5.107.72:46611: session id = 83b0e3c0ba724317832f8505a1ffb66e-08d82302438544e4925a4f511f13fb7e
I20240502 14:23:53.023666 6144 tablet_copy_source_session.cc:205] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:23:53.024639 9082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08d82302438544e4925a4f511f13fb7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:53.028605 9082 tablet_copy_client.cc:783] T 08d82302438544e4925a4f511f13fb7e P 83b0e3c0ba724317832f8505a1ffb66e: tablet copy: Starting download of 0 data blocks...
I20240502 14:23:53.028841 9082 tablet_copy_client.cc:647] T 08d82302438544e4925a4f511f13fb7e P 83b0e3c0ba724317832f8505a1ffb66e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:23:53.030310 9086 tablet_copy_client.cc:1002] T 08d82302438544e4925a4f511f13fb7e P 83b0e3c0ba724317832f8505a1ffb66e: tablet copy: Injecting latency into file download: 1500
W20240502 14:23:53.045269 5927 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:23:53.046164 5927 consensus_peers.cc:485] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:23:53.316496 8946 tablet_copy_client.cc:523] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:23:53.319514 8946 tablet_bootstrap.cc:492] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1: Bootstrap starting.
I20240502 14:23:53.350397 8946 tablet_bootstrap.cc:492] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:23:53.350831 8946 tablet_bootstrap.cc:492] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1: Bootstrap complete.
I20240502 14:23:53.351081 8946 ts_tablet_manager.cc:1367] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1: Time spent bootstrapping tablet: real 0.032s user 0.028s sys 0.004s
I20240502 14:23:53.351817 8946 raft_consensus.cc:348] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } }
I20240502 14:23:53.352092 8946 raft_consensus.cc:729] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 806206bb049040bf8db48a7d8d3840c1, State: Initialized, Role: LEARNER
I20240502 14:23:53.352308 8946 consensus_queue.cc:260] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } }
I20240502 14:23:53.353170 8946 ts_tablet_manager.cc:1398] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:23:53.353962 6144 tablet_copy_service.cc:342] P 333acff728274715a9816c286921ddd0: Request end of tablet copy session 806206bb049040bf8db48a7d8d3840c1-9fdda89a7743466a8aeceb543124fd79 received from {username='slave'} at 127.5.107.70:36379
I20240502 14:23:53.354106 6144 tablet_copy_service.cc:433] P 333acff728274715a9816c286921ddd0: ending tablet copy session 806206bb049040bf8db48a7d8d3840c1-9fdda89a7743466a8aeceb543124fd79 on tablet 9fdda89a7743466a8aeceb543124fd79 with peer 806206bb049040bf8db48a7d8d3840c1
W20240502 14:23:53.423216 6057 consensus_peers.cc:485] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:23:53.457983 7374 raft_consensus.cc:1203] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240502 14:23:53.537631 5797 consensus_peers.cc:485] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:23:53.862715 9125 raft_consensus.cc:1050] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0: attempting to promote NON_VOTER 806206bb049040bf8db48a7d8d3840c1 to VOTER
I20240502 14:23:53.863395 9125 consensus_queue.cc:237] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } }
I20240502 14:23:53.865643 7374 raft_consensus.cc:1261] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1 [term 1 LEARNER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:23:53.865705 7504 raft_consensus.cc:1261] T 9fdda89a7743466a8aeceb543124fd79 P 681fd3c49e914094bf28a7df155cffc9 [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:23:53.865746 5992 raft_consensus.cc:1261] T 9fdda89a7743466a8aeceb543124fd79 P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:23:53.866219 9142 consensus_queue.cc:1035] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:23:53.866585 9125 consensus_queue.cc:1035] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:23:53.866865 9105 consensus_queue.cc:1035] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:23:53.870165 9105 raft_consensus.cc:2932] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:23:53.870718 5992 raft_consensus.cc:2932] T 9fdda89a7743466a8aeceb543124fd79 P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:23:53.873432 7504 raft_consensus.cc:2932] T 9fdda89a7743466a8aeceb543124fd79 P 681fd3c49e914094bf28a7df155cffc9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:23:53.874574 7374 raft_consensus.cc:2932] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:23:53.878669 5586 catalog_manager.cc:5484] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1 reported cstate change: config changed from index 8 to 9, 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:23:53.907616 6122 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9fdda89a7743466a8aeceb543124fd79"
dest_uuid: "333acff728274715a9816c286921ddd0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44010
I20240502 14:23:53.907827 6122 raft_consensus.cc:595] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:23:54.140717 6122 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9fdda89a7743466a8aeceb543124fd79"
dest_uuid: "333acff728274715a9816c286921ddd0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44038
I20240502 14:23:54.140909 6122 raft_consensus.cc:595] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [term 1 LEADER]: Received request to transfer leadership
W20240502 14:23:54.177793 5927 consensus_peers.cc:485] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:23:54.350857 9125 raft_consensus.cc:979] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0: : Instructing follower 806206bb049040bf8db48a7d8d3840c1 to start an election
I20240502 14:23:54.351066 9125 raft_consensus.cc:1067] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [term 1 LEADER]: Signalling peer 806206bb049040bf8db48a7d8d3840c1 to start an election
I20240502 14:23:54.351907 7374 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9fdda89a7743466a8aeceb543124fd79"
dest_uuid: "806206bb049040bf8db48a7d8d3840c1"
from {username='slave'} at 127.5.107.68:33599
I20240502 14:23:54.352247 7374 raft_consensus.cc:482] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:23:54.352422 7374 raft_consensus.cc:3037] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:23:54.354805 7374 raft_consensus.cc:504] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } }
I20240502 14:23:54.358760 6122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdda89a7743466a8aeceb543124fd79" candidate_uuid: "806206bb049040bf8db48a7d8d3840c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "333acff728274715a9816c286921ddd0"
I20240502 14:23:54.359027 6122 raft_consensus.cc:3032] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:23:54.359145 6122 raft_consensus.cc:729] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 333acff728274715a9816c286921ddd0, State: Running, Role: LEADER
I20240502 14:23:54.359493 6122 consensus_queue.cc:260] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } }
I20240502 14:23:54.360025 6122 raft_consensus.cc:3037] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:23:54.366454 7374 leader_election.cc:288] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1 [CANDIDATE]: Term 2 election: Requested vote from peers 333acff728274715a9816c286921ddd0 (127.5.107.68:45913), 181b466de0084fcbace0ca327fb1cbef (127.5.107.67:41517), 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71:42329)
I20240502 14:23:54.369475 6122 raft_consensus.cc:2454] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 806206bb049040bf8db48a7d8d3840c1 in term 2.
I20240502 14:23:54.396610 7504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdda89a7743466a8aeceb543124fd79" candidate_uuid: "806206bb049040bf8db48a7d8d3840c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "681fd3c49e914094bf28a7df155cffc9"
I20240502 14:23:54.397020 7504 raft_consensus.cc:3037] T 9fdda89a7743466a8aeceb543124fd79 P 681fd3c49e914094bf28a7df155cffc9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:23:54.400594 7504 raft_consensus.cc:2454] T 9fdda89a7743466a8aeceb543124fd79 P 681fd3c49e914094bf28a7df155cffc9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 806206bb049040bf8db48a7d8d3840c1 in term 2.
I20240502 14:23:54.401094 7310 leader_election.cc:302] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 333acff728274715a9816c286921ddd0, 681fd3c49e914094bf28a7df155cffc9, 806206bb049040bf8db48a7d8d3840c1; no voters:
I20240502 14:23:54.401576 5992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdda89a7743466a8aeceb543124fd79" candidate_uuid: "806206bb049040bf8db48a7d8d3840c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "181b466de0084fcbace0ca327fb1cbef"
I20240502 14:23:54.401861 5992 raft_consensus.cc:3037] T 9fdda89a7743466a8aeceb543124fd79 P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:23:54.404749 5992 raft_consensus.cc:2454] T 9fdda89a7743466a8aeceb543124fd79 P 181b466de0084fcbace0ca327fb1cbef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 806206bb049040bf8db48a7d8d3840c1 in term 2.
I20240502 14:23:54.405728 9200 raft_consensus.cc:2781] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:23:54.406833 9200 raft_consensus.cc:686] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1 [term 2 LEADER]: Becoming Leader. State: Replica: 806206bb049040bf8db48a7d8d3840c1, State: Running, Role: LEADER
I20240502 14:23:54.407640 9200 consensus_queue.cc:237] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } }
I20240502 14:23:54.411688 5592 catalog_manager.cc:5484] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1 reported cstate change: term changed from 1 to 2, leader changed from 333acff728274715a9816c286921ddd0 (127.5.107.68) to 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70). New cstate: current_term: 2 leader_uuid: "806206bb049040bf8db48a7d8d3840c1" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 14:23:54.434787 5576 catalog_manager.cc:4573] TS 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7d944a63fdf24ecd8fadbad3b620af0a: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
W20240502 14:23:54.444986 5797 consensus_peers.cc:485] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:23:54.445529 9059 tablet_copy_client.cc:523] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:23:54.447811 9059 tablet_bootstrap.cc:492] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2: Bootstrap starting.
I20240502 14:23:54.472959 9059 tablet_bootstrap.cc:492] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:23:54.473397 9059 tablet_bootstrap.cc:492] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2: Bootstrap complete.
I20240502 14:23:54.473640 9059 ts_tablet_manager.cc:1367] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2: Time spent bootstrapping tablet: real 0.026s user 0.024s sys 0.004s
I20240502 14:23:54.474274 9059 raft_consensus.cc:348] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: NON_VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: true } }
I20240502 14:23:54.474457 9059 raft_consensus.cc:729] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3416b273466c495298342b6e892015c2, State: Initialized, Role: LEARNER
I20240502 14:23:54.474592 9059 consensus_queue.cc:260] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: NON_VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: true } }
I20240502 14:23:54.475289 9059 ts_tablet_manager.cc:1398] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:23:54.476078 5884 tablet_copy_service.cc:342] P 83e62d4e5f8a4316bc1bcb8c18458081: Request end of tablet copy session 3416b273466c495298342b6e892015c2-81cfc8c8608b4cb0b7c303a910791d4b received from {username='slave'} at 127.5.107.69:48521
I20240502 14:23:54.476260 5884 tablet_copy_service.cc:433] P 83e62d4e5f8a4316bc1bcb8c18458081: ending tablet copy session 3416b273466c495298342b6e892015c2-81cfc8c8608b4cb0b7c303a910791d4b on tablet 81cfc8c8608b4cb0b7c303a910791d4b with peer 3416b273466c495298342b6e892015c2
I20240502 14:23:54.530658 9082 tablet_copy_client.cc:523] T 08d82302438544e4925a4f511f13fb7e P 83b0e3c0ba724317832f8505a1ffb66e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:23:54.533324 9082 tablet_bootstrap.cc:492] T 08d82302438544e4925a4f511f13fb7e P 83b0e3c0ba724317832f8505a1ffb66e: Bootstrap starting.
I20240502 14:23:54.560873 9082 tablet_bootstrap.cc:492] T 08d82302438544e4925a4f511f13fb7e P 83b0e3c0ba724317832f8505a1ffb66e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:23:54.561259 9082 tablet_bootstrap.cc:492] T 08d82302438544e4925a4f511f13fb7e P 83b0e3c0ba724317832f8505a1ffb66e: Bootstrap complete.
I20240502 14:23:54.561514 9082 ts_tablet_manager.cc:1367] T 08d82302438544e4925a4f511f13fb7e P 83b0e3c0ba724317832f8505a1ffb66e: Time spent bootstrapping tablet: real 0.028s user 0.026s sys 0.004s
I20240502 14:23:54.562341 9082 raft_consensus.cc:348] T 08d82302438544e4925a4f511f13fb7e P 83b0e3c0ba724317832f8505a1ffb66e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } }
I20240502 14:23:54.562570 9082 raft_consensus.cc:729] T 08d82302438544e4925a4f511f13fb7e P 83b0e3c0ba724317832f8505a1ffb66e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83b0e3c0ba724317832f8505a1ffb66e, State: Initialized, Role: LEARNER
I20240502 14:23:54.562794 9082 consensus_queue.cc:260] T 08d82302438544e4925a4f511f13fb7e P 83b0e3c0ba724317832f8505a1ffb66e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } }
I20240502 14:23:54.563445 9082 ts_tablet_manager.cc:1398] T 08d82302438544e4925a4f511f13fb7e P 83b0e3c0ba724317832f8505a1ffb66e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:23:54.564193 6144 tablet_copy_service.cc:342] P 333acff728274715a9816c286921ddd0: Request end of tablet copy session 83b0e3c0ba724317832f8505a1ffb66e-08d82302438544e4925a4f511f13fb7e received from {username='slave'} at 127.5.107.72:46611
I20240502 14:23:54.564333 6144 tablet_copy_service.cc:433] P 333acff728274715a9816c286921ddd0: ending tablet copy session 83b0e3c0ba724317832f8505a1ffb66e-08d82302438544e4925a4f511f13fb7e on tablet 08d82302438544e4925a4f511f13fb7e with peer 83b0e3c0ba724317832f8505a1ffb66e
I20240502 14:23:54.590975 7634 raft_consensus.cc:1203] T 08d82302438544e4925a4f511f13fb7e P 83b0e3c0ba724317832f8505a1ffb66e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:23:54.913628 7242 raft_consensus.cc:1203] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:23:54.934319 7504 raft_consensus.cc:1261] T 9fdda89a7743466a8aeceb543124fd79 P 681fd3c49e914094bf28a7df155cffc9 [term 2 FOLLOWER]: Refusing update from remote peer 806206bb049040bf8db48a7d8d3840c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:23:54.935027 9252 consensus_queue.cc:1035] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:23:54.987083 6122 raft_consensus.cc:1261] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Refusing update from remote peer 806206bb049040bf8db48a7d8d3840c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:23:54.987548 9252 consensus_queue.cc:1035] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:23:54.992542 5992 raft_consensus.cc:1261] T 9fdda89a7743466a8aeceb543124fd79 P 181b466de0084fcbace0ca327fb1cbef [term 2 FOLLOWER]: Refusing update from remote peer 806206bb049040bf8db48a7d8d3840c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:23:54.993089 9258 consensus_queue.cc:1035] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 14:23:55.101656 6057 consensus_peers.cc:485] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:23:55.123118 9125 raft_consensus.cc:1050] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0: attempting to promote NON_VOTER 83b0e3c0ba724317832f8505a1ffb66e to VOTER
I20240502 14:23:55.124013 9125 consensus_queue.cc:237] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } }
I20240502 14:23:55.126397 7634 raft_consensus.cc:1261] T 08d82302438544e4925a4f511f13fb7e P 83b0e3c0ba724317832f8505a1ffb66e [term 1 LEARNER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:23:55.126550 5992 raft_consensus.cc:1261] T 08d82302438544e4925a4f511f13fb7e P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:23:55.126585 5864 raft_consensus.cc:1261] T 08d82302438544e4925a4f511f13fb7e P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:23:55.127182 9280 consensus_queue.cc:1035] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:23:55.127537 9125 consensus_queue.cc:1035] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:23:55.128268 9278 consensus_queue.cc:1035] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:23:55.132459 7634 raft_consensus.cc:2932] T 08d82302438544e4925a4f511f13fb7e P 83b0e3c0ba724317832f8505a1ffb66e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:23:55.132891 5992 raft_consensus.cc:2932] T 08d82302438544e4925a4f511f13fb7e P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:23:55.131922 9280 raft_consensus.cc:2932] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:23:55.137039 5592 catalog_manager.cc:5484] T 08d82302438544e4925a4f511f13fb7e P 181b466de0084fcbace0ca327fb1cbef reported cstate change: config changed from index 4 to 5, 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:23:55.140877 5864 raft_consensus.cc:2932] T 08d82302438544e4925a4f511f13fb7e P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:23:55.143479 6122 consensus_queue.cc:237] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } }
I20240502 14:23:55.144989 7634 raft_consensus.cc:1261] T 08d82302438544e4925a4f511f13fb7e P 83b0e3c0ba724317832f8505a1ffb66e [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:23:55.145563 5992 raft_consensus.cc:1261] T 08d82302438544e4925a4f511f13fb7e P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:23:55.145545 9280 consensus_queue.cc:1035] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:23:55.146075 9142 consensus_queue.cc:1035] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:23:55.148308 9280 raft_consensus.cc:2932] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:23:55.149114 7634 raft_consensus.cc:2932] T 08d82302438544e4925a4f511f13fb7e P 83b0e3c0ba724317832f8505a1ffb66e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:23:55.150203 5992 raft_consensus.cc:2932] T 08d82302438544e4925a4f511f13fb7e P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:23:55.152247 5578 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 08d82302438544e4925a4f511f13fb7e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:23:55.153431 5583 catalog_manager.cc:5484] T 08d82302438544e4925a4f511f13fb7e P 83b0e3c0ba724317832f8505a1ffb66e reported cstate change: config changed from index 5 to 6, VOTER 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66) evicted. New cstate: current_term: 1 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:23:55.156661 5844 tablet_service.cc:1514] Processing DeleteTablet for tablet 08d82302438544e4925a4f511f13fb7e with delete_type TABLET_DATA_TOMBSTONED (TS 83e62d4e5f8a4316bc1bcb8c18458081 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48002
I20240502 14:23:55.157626 9288 tablet_replica.cc:330] T 08d82302438544e4925a4f511f13fb7e P 83e62d4e5f8a4316bc1bcb8c18458081: stopping tablet replica
I20240502 14:23:55.158071 9288 raft_consensus.cc:2229] T 08d82302438544e4925a4f511f13fb7e P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:23:55.158412 9288 raft_consensus.cc:2258] T 08d82302438544e4925a4f511f13fb7e P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:23:55.159986 9288 ts_tablet_manager.cc:1875] T 08d82302438544e4925a4f511f13fb7e P 83e62d4e5f8a4316bc1bcb8c18458081: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:23:55.164379 9288 ts_tablet_manager.cc:1888] T 08d82302438544e4925a4f511f13fb7e P 83e62d4e5f8a4316bc1bcb8c18458081: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:23:55.164510 9288 log.cc:1196] T 08d82302438544e4925a4f511f13fb7e P 83e62d4e5f8a4316bc1bcb8c18458081: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/wals/08d82302438544e4925a4f511f13fb7e
I20240502 14:23:55.165273 5577 catalog_manager.cc:4830] TS 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015): tablet 08d82302438544e4925a4f511f13fb7e (table rebalance_test_table_1 [id=a3f5812bc1aa47c1990b9a769141e329]) successfully deleted
I20240502 14:23:55.349138 6122 consensus_queue.cc:237] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } }
W20240502 14:23:55.351366 6057 consensus_peers.cc:485] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 -> Peer 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72:35323): Couldn't send request to peer 83b0e3c0ba724317832f8505a1ffb66e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d944a63fdf24ecd8fadbad3b620af0a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:23:55.351331 7373 raft_consensus.cc:1261] T 7d944a63fdf24ecd8fadbad3b620af0a P 806206bb049040bf8db48a7d8d3840c1 [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:23:55.351502 5864 raft_consensus.cc:1261] T 7d944a63fdf24ecd8fadbad3b620af0a P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:23:55.351868 9142 consensus_queue.cc:1035] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:23:55.352190 9280 consensus_queue.cc:1035] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:23:55.354547 9278 raft_consensus.cc:2932] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } } }
I20240502 14:23:55.355094 7373 raft_consensus.cc:2932] T 7d944a63fdf24ecd8fadbad3b620af0a P 806206bb049040bf8db48a7d8d3840c1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } } }
I20240502 14:23:55.355194 5864 raft_consensus.cc:2932] T 7d944a63fdf24ecd8fadbad3b620af0a P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } } }
I20240502 14:23:55.363090 5592 catalog_manager.cc:5484] T 7d944a63fdf24ecd8fadbad3b620af0a P 806206bb049040bf8db48a7d8d3840c1 reported cstate change: config changed from index 7 to 8, NON_VOTER 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) added. New cstate: current_term: 1 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } } }
I20240502 14:23:55.369580 9312 raft_consensus.cc:1050] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081: attempting to promote NON_VOTER 3416b273466c495298342b6e892015c2 to VOTER
I20240502 14:23:55.370503 9312 consensus_queue.cc:237] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } }
I20240502 14:23:55.373216 7242 raft_consensus.cc:1261] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2 [term 1 LEARNER]: Refusing update from remote peer 83e62d4e5f8a4316bc1bcb8c18458081: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:23:55.373436 5992 raft_consensus.cc:1261] T 81cfc8c8608b4cb0b7c303a910791d4b P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Refusing update from remote peer 83e62d4e5f8a4316bc1bcb8c18458081: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:23:55.373677 6122 raft_consensus.cc:1261] T 81cfc8c8608b4cb0b7c303a910791d4b P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Refusing update from remote peer 83e62d4e5f8a4316bc1bcb8c18458081: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:23:55.373878 9312 consensus_queue.cc:1035] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:23:55.374253 9121 consensus_queue.cc:1035] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:23:55.374728 9313 consensus_queue.cc:1035] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:23:55.388800 9312 raft_consensus.cc:2932] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3416b273466c495298342b6e892015c2 (127.5.107.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } }
I20240502 14:23:55.389642 7242 raft_consensus.cc:2932] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3416b273466c495298342b6e892015c2 (127.5.107.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } }
I20240502 14:23:55.390717 6122 raft_consensus.cc:2932] T 81cfc8c8608b4cb0b7c303a910791d4b P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3416b273466c495298342b6e892015c2 (127.5.107.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } }
I20240502 14:23:55.394338 5992 raft_consensus.cc:2932] T 81cfc8c8608b4cb0b7c303a910791d4b P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3416b273466c495298342b6e892015c2 (127.5.107.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } }
I20240502 14:23:55.394718 5583 catalog_manager.cc:5484] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 reported cstate change: config changed from index 4 to 5, 3416b273466c495298342b6e892015c2 (127.5.107.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 14:23:55.399088 9121 consensus_queue.cc:1159] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "333acff728274715a9816c286921ddd0" responder_term: 1 status { last_received { term: 1 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 1 index: 5 } }
I20240502 14:23:55.400683 5864 consensus_queue.cc:237] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } }
I20240502 14:23:55.402343 7242 raft_consensus.cc:1261] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2 [term 1 FOLLOWER]: Refusing update from remote peer 83e62d4e5f8a4316bc1bcb8c18458081: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:23:55.403057 6122 raft_consensus.cc:1261] T 81cfc8c8608b4cb0b7c303a910791d4b P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Refusing update from remote peer 83e62d4e5f8a4316bc1bcb8c18458081: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:23:55.403553 9313 consensus_queue.cc:1035] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:23:55.404165 9121 consensus_queue.cc:1035] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:23:55.405937 9312 raft_consensus.cc:2932] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 181b466de0084fcbace0ca327fb1cbef (127.5.107.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } }
I20240502 14:23:55.406406 7242 raft_consensus.cc:2932] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 181b466de0084fcbace0ca327fb1cbef (127.5.107.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } }
I20240502 14:23:55.408324 5577 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 81cfc8c8608b4cb0b7c303a910791d4b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:23:55.409821 5592 catalog_manager.cc:5484] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 reported cstate change: config changed from index 5 to 6, VOTER 181b466de0084fcbace0ca327fb1cbef (127.5.107.67) evicted. New cstate: current_term: 1 leader_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:23:55.412923 5973 tablet_service.cc:1514] Processing DeleteTablet for tablet 81cfc8c8608b4cb0b7c303a910791d4b with delete_type TABLET_DATA_TOMBSTONED (TS 181b466de0084fcbace0ca327fb1cbef not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36800
I20240502 14:23:55.414263 9323 tablet_replica.cc:330] T 81cfc8c8608b4cb0b7c303a910791d4b P 181b466de0084fcbace0ca327fb1cbef: stopping tablet replica
I20240502 14:23:55.414499 9323 raft_consensus.cc:2229] T 81cfc8c8608b4cb0b7c303a910791d4b P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:23:55.414791 9323 raft_consensus.cc:2258] T 81cfc8c8608b4cb0b7c303a910791d4b P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:23:55.414620 6122 raft_consensus.cc:2932] T 81cfc8c8608b4cb0b7c303a910791d4b P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 181b466de0084fcbace0ca327fb1cbef (127.5.107.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } }
W20240502 14:23:55.417229 5927 consensus_peers.cc:485] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:23:55.418360 9323 ts_tablet_manager.cc:1875] T 81cfc8c8608b4cb0b7c303a910791d4b P 181b466de0084fcbace0ca327fb1cbef: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:23:55.424465 9323 ts_tablet_manager.cc:1888] T 81cfc8c8608b4cb0b7c303a910791d4b P 181b466de0084fcbace0ca327fb1cbef: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:23:55.424566 9323 log.cc:1196] T 81cfc8c8608b4cb0b7c303a910791d4b P 181b466de0084fcbace0ca327fb1cbef: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal/wals/81cfc8c8608b4cb0b7c303a910791d4b
I20240502 14:23:55.425285 5578 catalog_manager.cc:4830] TS 181b466de0084fcbace0ca327fb1cbef (127.5.107.67:41517): tablet 81cfc8c8608b4cb0b7c303a910791d4b (table rebalance_test_table_1 [id=a3f5812bc1aa47c1990b9a769141e329]) successfully deleted
I20240502 14:23:55.563938 7373 consensus_queue.cc:237] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } }
I20240502 14:23:55.565513 5992 raft_consensus.cc:1261] T 9fdda89a7743466a8aeceb543124fd79 P 181b466de0084fcbace0ca327fb1cbef [term 2 FOLLOWER]: Refusing update from remote peer 806206bb049040bf8db48a7d8d3840c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 14:23:55.565605 7504 raft_consensus.cc:1261] T 9fdda89a7743466a8aeceb543124fd79 P 681fd3c49e914094bf28a7df155cffc9 [term 2 FOLLOWER]: Refusing update from remote peer 806206bb049040bf8db48a7d8d3840c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 14:23:55.565995 9347 consensus_queue.cc:1035] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:23:55.566305 9346 consensus_queue.cc:1035] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:23:55.568156 9347 raft_consensus.cc:2932] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 333acff728274715a9816c286921ddd0 (127.5.107.68) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:23:55.568796 5993 raft_consensus.cc:2932] T 9fdda89a7743466a8aeceb543124fd79 P 181b466de0084fcbace0ca327fb1cbef [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 333acff728274715a9816c286921ddd0 (127.5.107.68) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:23:55.572827 7504 raft_consensus.cc:2932] T 9fdda89a7743466a8aeceb543124fd79 P 681fd3c49e914094bf28a7df155cffc9 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 333acff728274715a9816c286921ddd0 (127.5.107.68) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:23:55.576952 5578 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9fdda89a7743466a8aeceb543124fd79 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:23:55.577682 5591 catalog_manager.cc:5484] T 9fdda89a7743466a8aeceb543124fd79 P 681fd3c49e914094bf28a7df155cffc9 reported cstate change: config changed from index 9 to 11, VOTER 333acff728274715a9816c286921ddd0 (127.5.107.68) evicted. New cstate: current_term: 2 leader_uuid: "806206bb049040bf8db48a7d8d3840c1" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:23:55.580953 6103 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fdda89a7743466a8aeceb543124fd79 with delete_type TABLET_DATA_TOMBSTONED (TS 333acff728274715a9816c286921ddd0 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:44714
I20240502 14:23:55.581609 9350 tablet_replica.cc:330] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0: stopping tablet replica
I20240502 14:23:55.581871 9350 raft_consensus.cc:2229] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:23:55.582096 9350 raft_consensus.cc:2258] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:23:55.582829 9350 ts_tablet_manager.cc:1875] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:23:55.587218 9350 ts_tablet_manager.cc:1888] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240502 14:23:55.587324 9350 log.cc:1196] T 9fdda89a7743466a8aeceb543124fd79 P 333acff728274715a9816c286921ddd0: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal/wals/9fdda89a7743466a8aeceb543124fd79
I20240502 14:23:55.587900 5578 catalog_manager.cc:4830] TS 333acff728274715a9816c286921ddd0 (127.5.107.68:45913): tablet 9fdda89a7743466a8aeceb543124fd79 (table rebalance_test_table_2 [id=64688b21b92e4f8596609be0f50e11bf]) successfully deleted
W20240502 14:23:55.646922 6057 consensus_peers.cc:485] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:23:55.966233 9371 ts_tablet_manager.cc:902] T 7d944a63fdf24ecd8fadbad3b620af0a P 83b0e3c0ba724317832f8505a1ffb66e: Initiating tablet copy from peer 333acff728274715a9816c286921ddd0 (127.5.107.68:45913)
I20240502 14:23:55.967406 9371 tablet_copy_client.cc:308] T 7d944a63fdf24ecd8fadbad3b620af0a P 83b0e3c0ba724317832f8505a1ffb66e: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.68:45913
I20240502 14:23:55.968010 6144 tablet_copy_service.cc:140] P 333acff728274715a9816c286921ddd0: Received BeginTabletCopySession request for tablet 7d944a63fdf24ecd8fadbad3b620af0a from peer 83b0e3c0ba724317832f8505a1ffb66e ({username='slave'} at 127.5.107.72:46611)
I20240502 14:23:55.968154 6144 tablet_copy_service.cc:161] P 333acff728274715a9816c286921ddd0: Beginning new tablet copy session on tablet 7d944a63fdf24ecd8fadbad3b620af0a from peer 83b0e3c0ba724317832f8505a1ffb66e at {username='slave'} at 127.5.107.72:46611: session id = 83b0e3c0ba724317832f8505a1ffb66e-7d944a63fdf24ecd8fadbad3b620af0a
I20240502 14:23:55.970077 6144 tablet_copy_source_session.cc:205] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:23:55.971199 9371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d944a63fdf24ecd8fadbad3b620af0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:55.975198 9371 tablet_copy_client.cc:783] T 7d944a63fdf24ecd8fadbad3b620af0a P 83b0e3c0ba724317832f8505a1ffb66e: tablet copy: Starting download of 0 data blocks...
I20240502 14:23:55.975437 9371 tablet_copy_client.cc:647] T 7d944a63fdf24ecd8fadbad3b620af0a P 83b0e3c0ba724317832f8505a1ffb66e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:23:55.976670 9375 tablet_copy_client.cc:1002] T 7d944a63fdf24ecd8fadbad3b620af0a P 83b0e3c0ba724317832f8505a1ffb66e: tablet copy: Injecting latency into file download: 1500
W20240502 14:23:56.127807 5797 consensus_peers.cc:485] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:23:56.159801 5576 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240502 14:23:56.160899 5576 catalog_manager.cc:4573] TS 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet aca9200e5c264c27833d7a4a9122c4d9: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
W20240502 14:23:56.171172 6057 consensus_peers.cc:485] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:23:56.685133 5927 consensus_peers.cc:485] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:23:56.874341 5797 consensus_peers.cc:485] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:23:57.476465 5576 catalog_manager.cc:4573] TS 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 25f9ebd715dc4f96901b60796ea846ad: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
I20240502 14:23:57.477017 9371 tablet_copy_client.cc:523] T 7d944a63fdf24ecd8fadbad3b620af0a P 83b0e3c0ba724317832f8505a1ffb66e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:23:57.479496 9371 tablet_bootstrap.cc:492] T 7d944a63fdf24ecd8fadbad3b620af0a P 83b0e3c0ba724317832f8505a1ffb66e: Bootstrap starting.
I20240502 14:23:57.507989 9371 tablet_bootstrap.cc:492] T 7d944a63fdf24ecd8fadbad3b620af0a P 83b0e3c0ba724317832f8505a1ffb66e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:23:57.508412 9371 tablet_bootstrap.cc:492] T 7d944a63fdf24ecd8fadbad3b620af0a P 83b0e3c0ba724317832f8505a1ffb66e: Bootstrap complete.
I20240502 14:23:57.508709 9371 ts_tablet_manager.cc:1367] T 7d944a63fdf24ecd8fadbad3b620af0a P 83b0e3c0ba724317832f8505a1ffb66e: Time spent bootstrapping tablet: real 0.029s user 0.023s sys 0.005s
I20240502 14:23:57.509418 9371 raft_consensus.cc:348] T 7d944a63fdf24ecd8fadbad3b620af0a P 83b0e3c0ba724317832f8505a1ffb66e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } }
I20240502 14:23:57.509630 9371 raft_consensus.cc:729] T 7d944a63fdf24ecd8fadbad3b620af0a P 83b0e3c0ba724317832f8505a1ffb66e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83b0e3c0ba724317832f8505a1ffb66e, State: Initialized, Role: LEARNER
I20240502 14:23:57.509827 9371 consensus_queue.cc:260] T 7d944a63fdf24ecd8fadbad3b620af0a P 83b0e3c0ba724317832f8505a1ffb66e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } }
I20240502 14:23:57.510668 9371 ts_tablet_manager.cc:1398] T 7d944a63fdf24ecd8fadbad3b620af0a P 83b0e3c0ba724317832f8505a1ffb66e: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240502 14:23:57.511410 6144 tablet_copy_service.cc:342] P 333acff728274715a9816c286921ddd0: Request end of tablet copy session 83b0e3c0ba724317832f8505a1ffb66e-7d944a63fdf24ecd8fadbad3b620af0a received from {username='slave'} at 127.5.107.72:46611
I20240502 14:23:57.511555 6144 tablet_copy_service.cc:433] P 333acff728274715a9816c286921ddd0: ending tablet copy session 83b0e3c0ba724317832f8505a1ffb66e-7d944a63fdf24ecd8fadbad3b620af0a on tablet 7d944a63fdf24ecd8fadbad3b620af0a with peer 83b0e3c0ba724317832f8505a1ffb66e
W20240502 14:23:57.679296 6057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 27 similar messages]
W20240502 14:23:57.684281 6057 consensus_peers.cc:485] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:23:57.737900 5927 consensus_peers.cc:485] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:23:57.856068 7634 raft_consensus.cc:1203] T 7d944a63fdf24ecd8fadbad3b620af0a P 83b0e3c0ba724317832f8505a1ffb66e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240502 14:23:58.024372 5797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:23:58.148195 5927 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 14:23:58.225903 6057 consensus_peers.cc:485] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:23:58.309643 9443 raft_consensus.cc:1050] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0: attempting to promote NON_VOTER 83b0e3c0ba724317832f8505a1ffb66e to VOTER
I20240502 14:23:58.310379 9443 consensus_queue.cc:237] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } }
I20240502 14:23:58.312490 7634 raft_consensus.cc:1261] T 7d944a63fdf24ecd8fadbad3b620af0a P 83b0e3c0ba724317832f8505a1ffb66e [term 1 LEARNER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:23:58.312707 7373 raft_consensus.cc:1261] T 7d944a63fdf24ecd8fadbad3b620af0a P 806206bb049040bf8db48a7d8d3840c1 [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:23:58.312997 5864 raft_consensus.cc:1261] T 7d944a63fdf24ecd8fadbad3b620af0a P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:23:58.313292 9443 consensus_queue.cc:1035] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:23:58.313620 9389 consensus_queue.cc:1035] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:23:58.313910 9445 consensus_queue.cc:1035] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:23:58.322247 9443 raft_consensus.cc:2932] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:23:58.322865 7373 raft_consensus.cc:2932] T 7d944a63fdf24ecd8fadbad3b620af0a P 806206bb049040bf8db48a7d8d3840c1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:23:58.323145 7634 raft_consensus.cc:2932] T 7d944a63fdf24ecd8fadbad3b620af0a P 83b0e3c0ba724317832f8505a1ffb66e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:23:58.326221 5864 raft_consensus.cc:2932] T 7d944a63fdf24ecd8fadbad3b620af0a P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:23:58.327493 5591 catalog_manager.cc:5484] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 reported cstate change: config changed from index 8 to 9, 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:23:58.333396 6122 consensus_queue.cc:237] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } }
I20240502 14:23:58.334782 7634 raft_consensus.cc:1261] T 7d944a63fdf24ecd8fadbad3b620af0a P 83b0e3c0ba724317832f8505a1ffb66e [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:23:58.335422 9443 consensus_queue.cc:1035] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:23:58.337469 7373 raft_consensus.cc:1261] T 7d944a63fdf24ecd8fadbad3b620af0a P 806206bb049040bf8db48a7d8d3840c1 [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:23:58.337980 9445 consensus_queue.cc:1035] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:23:58.337798 9389 raft_consensus.cc:2932] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:23:58.339377 7373 raft_consensus.cc:2932] T 7d944a63fdf24ecd8fadbad3b620af0a P 806206bb049040bf8db48a7d8d3840c1 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:23:58.341306 5578 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7d944a63fdf24ecd8fadbad3b620af0a with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:23:58.341794 7634 raft_consensus.cc:2932] T 7d944a63fdf24ecd8fadbad3b620af0a P 83b0e3c0ba724317832f8505a1ffb66e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:23:58.342590 5592 catalog_manager.cc:5484] T 7d944a63fdf24ecd8fadbad3b620af0a P 333acff728274715a9816c286921ddd0 reported cstate change: config changed from index 9 to 10, VOTER 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66) evicted. New cstate: current_term: 1 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:23:58.346443 5844 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d944a63fdf24ecd8fadbad3b620af0a with delete_type TABLET_DATA_TOMBSTONED (TS 83e62d4e5f8a4316bc1bcb8c18458081 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:48002
I20240502 14:23:58.349572 9453 tablet_replica.cc:330] T 7d944a63fdf24ecd8fadbad3b620af0a P 83e62d4e5f8a4316bc1bcb8c18458081: stopping tablet replica
I20240502 14:23:58.349759 9453 raft_consensus.cc:2229] T 7d944a63fdf24ecd8fadbad3b620af0a P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:23:58.350122 9453 raft_consensus.cc:2258] T 7d944a63fdf24ecd8fadbad3b620af0a P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:23:58.350914 9453 ts_tablet_manager.cc:1875] T 7d944a63fdf24ecd8fadbad3b620af0a P 83e62d4e5f8a4316bc1bcb8c18458081: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:23:58.354884 9453 ts_tablet_manager.cc:1888] T 7d944a63fdf24ecd8fadbad3b620af0a P 83e62d4e5f8a4316bc1bcb8c18458081: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 14:23:58.355007 9453 log.cc:1196] T 7d944a63fdf24ecd8fadbad3b620af0a P 83e62d4e5f8a4316bc1bcb8c18458081: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/wals/7d944a63fdf24ecd8fadbad3b620af0a
I20240502 14:23:58.355747 5577 catalog_manager.cc:4830] TS 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015): tablet 7d944a63fdf24ecd8fadbad3b620af0a (table rebalance_test_table_0 [id=7a7408907bfc4fd2bd39cf4e8b4c97fb]) successfully deleted
I20240502 14:23:58.424065 6122 consensus_queue.cc:237] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } }
W20240502 14:23:58.426283 6057 consensus_peers.cc:485] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 -> Peer 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72:35323): Couldn't send request to peer 83b0e3c0ba724317832f8505a1ffb66e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25f9ebd715dc4f96901b60796ea846ad. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:23:58.426280 5864 raft_consensus.cc:1261] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:23:58.426417 7242 raft_consensus.cc:1261] T 25f9ebd715dc4f96901b60796ea846ad P 3416b273466c495298342b6e892015c2 [term 2 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:23:58.426748 9443 consensus_queue.cc:1035] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:23:58.426981 9389 consensus_queue.cc:1035] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:23:58.429363 9445 raft_consensus.cc:2932] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } } }
I20240502 14:23:58.429976 7242 raft_consensus.cc:2932] T 25f9ebd715dc4f96901b60796ea846ad P 3416b273466c495298342b6e892015c2 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } } }
I20240502 14:23:58.433413 5864 raft_consensus.cc:2932] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } } }
I20240502 14:23:58.434525 5583 catalog_manager.cc:5484] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 reported cstate change: config changed from index 8 to 9, NON_VOTER 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) added. New cstate: current_term: 2 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:23:58.438915 5797 consensus_peers.cc:485] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:23:58.723729 6057 consensus_peers.cc:485] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:23:58.876943 9483 ts_tablet_manager.cc:902] T 25f9ebd715dc4f96901b60796ea846ad P 83b0e3c0ba724317832f8505a1ffb66e: Initiating tablet copy from peer 333acff728274715a9816c286921ddd0 (127.5.107.68:45913)
I20240502 14:23:58.878118 9483 tablet_copy_client.cc:308] T 25f9ebd715dc4f96901b60796ea846ad P 83b0e3c0ba724317832f8505a1ffb66e: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.68:45913
I20240502 14:23:58.878715 6144 tablet_copy_service.cc:140] P 333acff728274715a9816c286921ddd0: Received BeginTabletCopySession request for tablet 25f9ebd715dc4f96901b60796ea846ad from peer 83b0e3c0ba724317832f8505a1ffb66e ({username='slave'} at 127.5.107.72:46611)
I20240502 14:23:58.878846 6144 tablet_copy_service.cc:161] P 333acff728274715a9816c286921ddd0: Beginning new tablet copy session on tablet 25f9ebd715dc4f96901b60796ea846ad from peer 83b0e3c0ba724317832f8505a1ffb66e at {username='slave'} at 127.5.107.72:46611: session id = 83b0e3c0ba724317832f8505a1ffb66e-25f9ebd715dc4f96901b60796ea846ad
I20240502 14:23:58.880728 6144 tablet_copy_source_session.cc:205] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:23:58.881810 9483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f9ebd715dc4f96901b60796ea846ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:23:58.885519 9483 tablet_copy_client.cc:783] T 25f9ebd715dc4f96901b60796ea846ad P 83b0e3c0ba724317832f8505a1ffb66e: tablet copy: Starting download of 0 data blocks...
I20240502 14:23:58.885735 9483 tablet_copy_client.cc:647] T 25f9ebd715dc4f96901b60796ea846ad P 83b0e3c0ba724317832f8505a1ffb66e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:23:58.887322 9487 tablet_copy_client.cc:1002] T 25f9ebd715dc4f96901b60796ea846ad P 83b0e3c0ba724317832f8505a1ffb66e: tablet copy: Injecting latency into file download: 1500
W20240502 14:23:58.997931 5927 consensus_peers.cc:485] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:23:59.369706 5797 consensus_peers.cc:485] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:23:59.818509 6057 consensus_peers.cc:485] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:23:59.848245 5927 consensus_peers.cc:485] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:24:00.387746 9483 tablet_copy_client.cc:523] T 25f9ebd715dc4f96901b60796ea846ad P 83b0e3c0ba724317832f8505a1ffb66e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:24:00.390200 9483 tablet_bootstrap.cc:492] T 25f9ebd715dc4f96901b60796ea846ad P 83b0e3c0ba724317832f8505a1ffb66e: Bootstrap starting.
I20240502 14:24:00.426541 9483 tablet_bootstrap.cc:492] T 25f9ebd715dc4f96901b60796ea846ad P 83b0e3c0ba724317832f8505a1ffb66e: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:24:00.426898 9483 tablet_bootstrap.cc:492] T 25f9ebd715dc4f96901b60796ea846ad P 83b0e3c0ba724317832f8505a1ffb66e: Bootstrap complete.
I20240502 14:24:00.427120 9483 ts_tablet_manager.cc:1367] T 25f9ebd715dc4f96901b60796ea846ad P 83b0e3c0ba724317832f8505a1ffb66e: Time spent bootstrapping tablet: real 0.037s user 0.035s sys 0.000s
I20240502 14:24:00.427676 9483 raft_consensus.cc:348] T 25f9ebd715dc4f96901b60796ea846ad P 83b0e3c0ba724317832f8505a1ffb66e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } }
I20240502 14:24:00.427871 9483 raft_consensus.cc:729] T 25f9ebd715dc4f96901b60796ea846ad P 83b0e3c0ba724317832f8505a1ffb66e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 83b0e3c0ba724317832f8505a1ffb66e, State: Initialized, Role: LEARNER
I20240502 14:24:00.428052 9483 consensus_queue.cc:260] T 25f9ebd715dc4f96901b60796ea846ad P 83b0e3c0ba724317832f8505a1ffb66e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } }
I20240502 14:24:00.428723 9483 ts_tablet_manager.cc:1398] T 25f9ebd715dc4f96901b60796ea846ad P 83b0e3c0ba724317832f8505a1ffb66e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:24:00.429404 6144 tablet_copy_service.cc:342] P 333acff728274715a9816c286921ddd0: Request end of tablet copy session 83b0e3c0ba724317832f8505a1ffb66e-25f9ebd715dc4f96901b60796ea846ad received from {username='slave'} at 127.5.107.72:46611
I20240502 14:24:00.429546 6144 tablet_copy_service.cc:433] P 333acff728274715a9816c286921ddd0: ending tablet copy session 83b0e3c0ba724317832f8505a1ffb66e-25f9ebd715dc4f96901b60796ea846ad on tablet 25f9ebd715dc4f96901b60796ea846ad with peer 83b0e3c0ba724317832f8505a1ffb66e
W20240502 14:24:00.482753 6057 consensus_peers.cc:485] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:24:00.879328 5797 consensus_peers.cc:485] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:24:00.904155 7633 raft_consensus.cc:1203] T 25f9ebd715dc4f96901b60796ea846ad P 83b0e3c0ba724317832f8505a1ffb66e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
W20240502 14:24:00.938422 5576 catalog_manager.cc:4573] TS 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9fdda89a7743466a8aeceb543124fd79: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
W20240502 14:24:01.052743 6057 consensus_peers.cc:485] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:24:01.312238 5927 consensus_peers.cc:485] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:24:01.510625 9559 raft_consensus.cc:1050] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0: attempting to promote NON_VOTER 83b0e3c0ba724317832f8505a1ffb66e to VOTER
I20240502 14:24:01.511554 9559 consensus_queue.cc:237] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } }
I20240502 14:24:01.514071 7633 raft_consensus.cc:1261] T 25f9ebd715dc4f96901b60796ea846ad P 83b0e3c0ba724317832f8505a1ffb66e [term 2 LEARNER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:24:01.514161 5864 raft_consensus.cc:1261] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:24:01.514225 7242 raft_consensus.cc:1261] T 25f9ebd715dc4f96901b60796ea846ad P 3416b273466c495298342b6e892015c2 [term 2 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:24:01.514791 9558 consensus_queue.cc:1035] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:24:01.515115 9561 consensus_queue.cc:1035] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:24:01.515434 9530 consensus_queue.cc:1035] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:24:01.518265 9530 raft_consensus.cc:2932] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:24:01.518965 7633 raft_consensus.cc:2932] T 25f9ebd715dc4f96901b60796ea846ad P 83b0e3c0ba724317832f8505a1ffb66e [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:24:01.518919 7242 raft_consensus.cc:2932] T 25f9ebd715dc4f96901b60796ea846ad P 3416b273466c495298342b6e892015c2 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:24:01.523025 5864 raft_consensus.cc:2932] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:24:01.523979 5586 catalog_manager.cc:5484] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 reported cstate change: config changed from index 9 to 10, 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:24:01.603931 6122 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "25f9ebd715dc4f96901b60796ea846ad"
dest_uuid: "333acff728274715a9816c286921ddd0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56848
I20240502 14:24:01.604169 6122 raft_consensus.cc:595] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [term 2 LEADER]: Received request to transfer leadership
W20240502 14:24:01.631906 5797 consensus_peers.cc:485] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:24:01.820434 6122 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "25f9ebd715dc4f96901b60796ea846ad"
dest_uuid: "333acff728274715a9816c286921ddd0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56860
I20240502 14:24:01.820645 6122 raft_consensus.cc:595] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [term 2 LEADER]: Received request to transfer leadership
I20240502 14:24:01.933555 9530 consensus_queue.cc:579] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 [LEADER]: Leader has been unable to successfully communicate with peer 1dbf482780eb47e8933983e4f0fd0444 for more than 30 seconds (30.275s)
I20240502 14:24:01.966428 9530 raft_consensus.cc:979] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0: : Instructing follower 83e62d4e5f8a4316bc1bcb8c18458081 to start an election
I20240502 14:24:01.966588 9530 raft_consensus.cc:1067] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [term 2 LEADER]: Signalling peer 83e62d4e5f8a4316bc1bcb8c18458081 to start an election
I20240502 14:24:01.967129 5864 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "25f9ebd715dc4f96901b60796ea846ad"
dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081"
from {username='slave'} at 127.5.107.68:59473
I20240502 14:24:01.967281 5864 raft_consensus.cc:482] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:24:01.967367 5864 raft_consensus.cc:3037] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:24:01.968998 5864 raft_consensus.cc:504] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } }
I20240502 14:24:01.969636 5864 leader_election.cc:288] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [CANDIDATE]: Term 3 election: Requested vote from peers 333acff728274715a9816c286921ddd0 (127.5.107.68:45913), 3416b273466c495298342b6e892015c2 (127.5.107.69:37277), 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72:35323)
I20240502 14:24:01.970016 6122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f9ebd715dc4f96901b60796ea846ad" candidate_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "333acff728274715a9816c286921ddd0"
I20240502 14:24:01.970259 6122 raft_consensus.cc:3032] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:24:01.970197 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f9ebd715dc4f96901b60796ea846ad" candidate_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "83b0e3c0ba724317832f8505a1ffb66e"
I20240502 14:24:01.970360 6122 raft_consensus.cc:729] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 333acff728274715a9816c286921ddd0, State: Running, Role: LEADER
I20240502 14:24:01.970293 7242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f9ebd715dc4f96901b60796ea846ad" candidate_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "3416b273466c495298342b6e892015c2"
I20240502 14:24:01.970444 7633 raft_consensus.cc:3037] T 25f9ebd715dc4f96901b60796ea846ad P 83b0e3c0ba724317832f8505a1ffb66e [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:24:01.970675 7242 raft_consensus.cc:3037] T 25f9ebd715dc4f96901b60796ea846ad P 3416b273466c495298342b6e892015c2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:24:01.970667 6122 consensus_queue.cc:260] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } }
I20240502 14:24:01.971019 6122 raft_consensus.cc:3037] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:24:01.972600 6122 raft_consensus.cc:2454] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e62d4e5f8a4316bc1bcb8c18458081 in term 3.
I20240502 14:24:01.972862 7633 raft_consensus.cc:2454] T 25f9ebd715dc4f96901b60796ea846ad P 83b0e3c0ba724317832f8505a1ffb66e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e62d4e5f8a4316bc1bcb8c18458081 in term 3.
I20240502 14:24:01.973309 5797 leader_election.cc:302] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 333acff728274715a9816c286921ddd0, 83b0e3c0ba724317832f8505a1ffb66e, 83e62d4e5f8a4316bc1bcb8c18458081; no voters:
I20240502 14:24:01.973394 7242 raft_consensus.cc:2454] T 25f9ebd715dc4f96901b60796ea846ad P 3416b273466c495298342b6e892015c2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e62d4e5f8a4316bc1bcb8c18458081 in term 3.
I20240502 14:24:01.973558 9395 raft_consensus.cc:2781] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:24:01.974023 9395 raft_consensus.cc:686] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [term 3 LEADER]: Becoming Leader. State: Replica: 83e62d4e5f8a4316bc1bcb8c18458081, State: Running, Role: LEADER
I20240502 14:24:01.974337 9395 consensus_queue.cc:237] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } }
I20240502 14:24:01.976464 5583 catalog_manager.cc:5484] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 reported cstate change: term changed from 2 to 3, leader changed from 333acff728274715a9816c286921ddd0 (127.5.107.68) to 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66). New cstate: current_term: 3 leader_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:24:02.159816 9591 consensus_queue.cc:579] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Leader has been unable to successfully communicate with peer 1dbf482780eb47e8933983e4f0fd0444 for more than 30 seconds (30.191s)
W20240502 14:24:02.331758 6057 consensus_peers.cc:485] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:24:02.510027 7633 raft_consensus.cc:1261] T 25f9ebd715dc4f96901b60796ea846ad P 83b0e3c0ba724317832f8505a1ffb66e [term 3 FOLLOWER]: Refusing update from remote peer 83e62d4e5f8a4316bc1bcb8c18458081: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 14:24:02.510673 9395 consensus_queue.cc:1035] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:24:02.564932 8922 consensus_queue.cc:579] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef [LEADER]: Leader has been unable to successfully communicate with peer 1dbf482780eb47e8933983e4f0fd0444 for more than 30 seconds (30.570s)
W20240502 14:24:02.566519 5927 consensus_peers.cc:485] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:24:02.567946 6122 raft_consensus.cc:1261] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [term 3 FOLLOWER]: Refusing update from remote peer 83e62d4e5f8a4316bc1bcb8c18458081: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 14:24:02.568446 9395 consensus_queue.cc:1035] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:24:02.575651 7242 raft_consensus.cc:1261] T 25f9ebd715dc4f96901b60796ea846ad P 3416b273466c495298342b6e892015c2 [term 3 FOLLOWER]: Refusing update from remote peer 83e62d4e5f8a4316bc1bcb8c18458081: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 14:24:02.576169 9395 consensus_queue.cc:1035] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:24:02.601774 9614 consensus_queue.cc:579] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Leader has been unable to successfully communicate with peer 1dbf482780eb47e8933983e4f0fd0444 for more than 30 seconds (30.412s)
I20240502 14:24:02.650211 9558 consensus_queue.cc:579] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 [LEADER]: Leader has been unable to successfully communicate with peer 1dbf482780eb47e8933983e4f0fd0444 for more than 30 seconds (30.320s)
W20240502 14:24:02.780656 6057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240502 14:24:03.045038 6057 consensus_peers.cc:485] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:24:03.174038 5864 consensus_queue.cc:237] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } }
I20240502 14:24:03.175609 7633 raft_consensus.cc:1261] T 25f9ebd715dc4f96901b60796ea846ad P 83b0e3c0ba724317832f8505a1ffb66e [term 3 FOLLOWER]: Refusing update from remote peer 83e62d4e5f8a4316bc1bcb8c18458081: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240502 14:24:03.175639 7242 raft_consensus.cc:1261] T 25f9ebd715dc4f96901b60796ea846ad P 3416b273466c495298342b6e892015c2 [term 3 FOLLOWER]: Refusing update from remote peer 83e62d4e5f8a4316bc1bcb8c18458081: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240502 14:24:03.176151 9637 consensus_queue.cc:1035] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:24:03.176425 9638 consensus_queue.cc:1035] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:24:03.178561 9614 raft_consensus.cc:2932] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER 333acff728274715a9816c286921ddd0 (127.5.107.68) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:24:03.179128 7242 raft_consensus.cc:2932] T 25f9ebd715dc4f96901b60796ea846ad P 3416b273466c495298342b6e892015c2 [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER 333acff728274715a9816c286921ddd0 (127.5.107.68) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:24:03.179196 7633 raft_consensus.cc:2932] T 25f9ebd715dc4f96901b60796ea846ad P 83b0e3c0ba724317832f8505a1ffb66e [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER 333acff728274715a9816c286921ddd0 (127.5.107.68) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:24:03.182946 5577 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 25f9ebd715dc4f96901b60796ea846ad with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:24:03.182813 5592 catalog_manager.cc:5484] T 25f9ebd715dc4f96901b60796ea846ad P 83e62d4e5f8a4316bc1bcb8c18458081 reported cstate change: config changed from index 10 to 12, VOTER 333acff728274715a9816c286921ddd0 (127.5.107.68) evicted. New cstate: current_term: 3 leader_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:24:03.187448 6103 tablet_service.cc:1514] Processing DeleteTablet for tablet 25f9ebd715dc4f96901b60796ea846ad with delete_type TABLET_DATA_TOMBSTONED (TS 333acff728274715a9816c286921ddd0 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:44714
I20240502 14:24:03.188128 9645 tablet_replica.cc:330] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0: stopping tablet replica
I20240502 14:24:03.188406 9645 raft_consensus.cc:2229] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 14:24:03.188834 9645 raft_consensus.cc:2258] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:24:03.189987 9645 ts_tablet_manager.cc:1875] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:24:03.194872 9645 ts_tablet_manager.cc:1888] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.11
I20240502 14:24:03.195021 9645 log.cc:1196] T 25f9ebd715dc4f96901b60796ea846ad P 333acff728274715a9816c286921ddd0: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal/wals/25f9ebd715dc4f96901b60796ea846ad
I20240502 14:24:03.195747 5578 catalog_manager.cc:4830] TS 333acff728274715a9816c286921ddd0 (127.5.107.68:45913): tablet 25f9ebd715dc4f96901b60796ea846ad (table rebalance_test_table_2 [id=64688b21b92e4f8596609be0f50e11bf]) successfully deleted
W20240502 14:24:03.374531 5797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 14:24:03.375581 5797 consensus_peers.cc:485] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:24:03.384792 5973 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:24:03.386480 7222 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:03.395087 6103 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:24:03.404422 7484 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:03.410261 5844 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:24:03.420189 7614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:03.427349 7354 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:24:03.433389 5927 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 14:24:03.508436 5864 consensus_queue.cc:237] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: NON_VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: true } }
I20240502 14:24:03.510838 7633 raft_consensus.cc:1261] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e [term 2 FOLLOWER]: Refusing update from remote peer 83e62d4e5f8a4316bc1bcb8c18458081: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:24:03.511061 6122 raft_consensus.cc:1261] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Refusing update from remote peer 83e62d4e5f8a4316bc1bcb8c18458081: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:24:03.511479 9637 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:24:03.511881 9638 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:24:03.514786 9614 raft_consensus.cc:2932] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: NON_VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: true } } }
I20240502 14:24:03.515282 6122 raft_consensus.cc:2932] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: NON_VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: true } } }
I20240502 14:24:03.519310 7633 raft_consensus.cc:2932] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: NON_VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: true } } }
W20240502 14:24:03.520558 5800 consensus_peers.cc:485] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71:42329): Couldn't send request to peer 681fd3c49e914094bf28a7df155cffc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5358f204e4324db78adad5af90c17b86. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:24:03.520960 5591 catalog_manager.cc:5484] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 reported cstate change: config changed from index 7 to 8, NON_VOTER 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) added. New cstate: current_term: 2 leader_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: NON_VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: true } } }
W20240502 14:24:03.576557 6057 consensus_peers.cc:485] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:24:03.645768 5927 consensus_peers.cc:485] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:24:03.869274 5797 consensus_peers.cc:485] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:24:03.888995 9614 consensus_queue.cc:579] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Leader has been unable to successfully communicate with peer 1dbf482780eb47e8933983e4f0fd0444 for more than 30 seconds (30.435s)
I20240502 14:24:04.015420 9712 ts_tablet_manager.cc:902] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9: Initiating tablet copy from peer 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015)
I20240502 14:24:04.016834 9712 tablet_copy_client.cc:308] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:34015
I20240502 14:24:04.023094 5884 tablet_copy_service.cc:140] P 83e62d4e5f8a4316bc1bcb8c18458081: Received BeginTabletCopySession request for tablet 5358f204e4324db78adad5af90c17b86 from peer 681fd3c49e914094bf28a7df155cffc9 ({username='slave'} at 127.5.107.71:41819)
I20240502 14:24:04.023265 5884 tablet_copy_service.cc:161] P 83e62d4e5f8a4316bc1bcb8c18458081: Beginning new tablet copy session on tablet 5358f204e4324db78adad5af90c17b86 from peer 681fd3c49e914094bf28a7df155cffc9 at {username='slave'} at 127.5.107.71:41819: session id = 681fd3c49e914094bf28a7df155cffc9-5358f204e4324db78adad5af90c17b86
I20240502 14:24:04.025719 5884 tablet_copy_source_session.cc:205] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:24:04.026916 9712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5358f204e4324db78adad5af90c17b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:04.031208 9712 tablet_copy_client.cc:783] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9: tablet copy: Starting download of 0 data blocks...
I20240502 14:24:04.031431 9712 tablet_copy_client.cc:647] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9: tablet copy: Starting download of 1 WAL segments...
I20240502 14:24:04.032796 9716 tablet_copy_client.cc:1002] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9: tablet copy: Injecting latency into file download: 1500
I20240502 14:24:04.125614 9530 consensus_queue.cc:579] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 [LEADER]: Leader has been unable to successfully communicate with peer 1dbf482780eb47e8933983e4f0fd0444 for more than 30 seconds (30.286s)
W20240502 14:24:04.395432 5576 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240502 14:24:04.396543 5576 catalog_manager.cc:4573] TS 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet aca9200e5c264c27833d7a4a9122c4d9: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
W20240502 14:24:04.624506 6057 consensus_peers.cc:485] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:24:04.910349 5927 consensus_peers.cc:485] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:24:05.533147 9712 tablet_copy_client.cc:523] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:24:05.535705 9712 tablet_bootstrap.cc:492] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9: Bootstrap starting.
I20240502 14:24:05.566515 9712 tablet_bootstrap.cc:492] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:24:05.566874 9712 tablet_bootstrap.cc:492] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9: Bootstrap complete.
I20240502 14:24:05.567099 9712 ts_tablet_manager.cc:1367] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9: Time spent bootstrapping tablet: real 0.031s user 0.022s sys 0.009s
I20240502 14:24:05.567672 9712 raft_consensus.cc:348] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: NON_VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: true } }
I20240502 14:24:05.567876 9712 raft_consensus.cc:729] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 681fd3c49e914094bf28a7df155cffc9, State: Initialized, Role: LEARNER
I20240502 14:24:05.568078 9712 consensus_queue.cc:260] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: NON_VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: true } }
I20240502 14:24:05.568708 9712 ts_tablet_manager.cc:1398] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20240502 14:24:05.569638 5884 tablet_copy_service.cc:342] P 83e62d4e5f8a4316bc1bcb8c18458081: Request end of tablet copy session 681fd3c49e914094bf28a7df155cffc9-5358f204e4324db78adad5af90c17b86 received from {username='slave'} at 127.5.107.71:41819
I20240502 14:24:05.569763 5884 tablet_copy_service.cc:433] P 83e62d4e5f8a4316bc1bcb8c18458081: ending tablet copy session 681fd3c49e914094bf28a7df155cffc9-5358f204e4324db78adad5af90c17b86 on tablet 5358f204e4324db78adad5af90c17b86 with peer 681fd3c49e914094bf28a7df155cffc9
I20240502 14:24:05.722014 7504 raft_consensus.cc:1203] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
W20240502 14:24:05.836323 6057 consensus_peers.cc:485] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:24:05.978780 6057 consensus_peers.cc:485] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:24:06.076506 5797 consensus_peers.cc:485] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:24:06.172724 5927 consensus_peers.cc:485] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:24:06.234607 9781 raft_consensus.cc:1050] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081: attempting to promote NON_VOTER 681fd3c49e914094bf28a7df155cffc9 to VOTER
I20240502 14:24:06.235306 9781 consensus_queue.cc:237] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } }
I20240502 14:24:06.237406 7504 raft_consensus.cc:1261] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 [term 2 LEARNER]: Refusing update from remote peer 83e62d4e5f8a4316bc1bcb8c18458081: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:24:06.237787 7633 raft_consensus.cc:1261] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e [term 2 FOLLOWER]: Refusing update from remote peer 83e62d4e5f8a4316bc1bcb8c18458081: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:24:06.237897 9780 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:24:06.238219 6122 raft_consensus.cc:1261] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Refusing update from remote peer 83e62d4e5f8a4316bc1bcb8c18458081: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:24:06.238796 9734 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:24:06.239149 9780 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:24:06.254261 9734 raft_consensus.cc:2932] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } }
I20240502 14:24:06.257000 6122 raft_consensus.cc:2932] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } }
I20240502 14:24:06.261428 5583 catalog_manager.cc:5484] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 reported cstate change: config changed from index 8 to 9, 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } }
I20240502 14:24:06.261392 7504 raft_consensus.cc:2932] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } }
I20240502 14:24:06.263681 7633 raft_consensus.cc:2932] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } }
I20240502 14:24:06.285483 5864 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5358f204e4324db78adad5af90c17b86"
dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41522
I20240502 14:24:06.285705 5864 raft_consensus.cc:595] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 LEADER]: Received request to transfer leadership
I20240502 14:24:06.500576 5864 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5358f204e4324db78adad5af90c17b86"
dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41536
I20240502 14:24:06.500802 5864 raft_consensus.cc:595] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 LEADER]: Received request to transfer leadership
W20240502 14:24:06.632304 5797 consensus_peers.cc:485] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:24:06.653846 9734 raft_consensus.cc:979] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081: : Instructing follower 333acff728274715a9816c286921ddd0 to start an election
I20240502 14:24:06.654027 9781 raft_consensus.cc:1067] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 LEADER]: Signalling peer 333acff728274715a9816c286921ddd0 to start an election
I20240502 14:24:06.654764 6122 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5358f204e4324db78adad5af90c17b86"
dest_uuid: "333acff728274715a9816c286921ddd0"
from {username='slave'} at 127.5.107.66:47943
I20240502 14:24:06.654994 6122 raft_consensus.cc:482] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:24:06.655134 6122 raft_consensus.cc:3037] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:24:06.657032 6122 raft_consensus.cc:504] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } }
I20240502 14:24:06.657934 6122 leader_election.cc:288] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 3 election: Requested vote from peers 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015), 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72:35323), 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71:42329)
I20240502 14:24:06.658084 5864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5358f204e4324db78adad5af90c17b86" candidate_uuid: "333acff728274715a9816c286921ddd0" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081"
I20240502 14:24:06.658289 5864 raft_consensus.cc:3032] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:24:06.658378 5864 raft_consensus.cc:729] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83e62d4e5f8a4316bc1bcb8c18458081, State: Running, Role: LEADER
I20240502 14:24:06.658380 7504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5358f204e4324db78adad5af90c17b86" candidate_uuid: "333acff728274715a9816c286921ddd0" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "681fd3c49e914094bf28a7df155cffc9"
I20240502 14:24:06.658669 7504 raft_consensus.cc:3037] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:24:06.658715 5864 consensus_queue.cc:260] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } }
I20240502 14:24:06.658792 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5358f204e4324db78adad5af90c17b86" candidate_uuid: "333acff728274715a9816c286921ddd0" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "83b0e3c0ba724317832f8505a1ffb66e"
I20240502 14:24:06.659025 7633 raft_consensus.cc:3037] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:24:06.659123 5864 raft_consensus.cc:3037] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:24:06.660984 5864 raft_consensus.cc:2454] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 333acff728274715a9816c286921ddd0 in term 3.
I20240502 14:24:06.661552 7504 raft_consensus.cc:2454] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 333acff728274715a9816c286921ddd0 in term 3.
I20240502 14:24:06.661988 6060 leader_election.cc:302] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 333acff728274715a9816c286921ddd0, 681fd3c49e914094bf28a7df155cffc9, 83e62d4e5f8a4316bc1bcb8c18458081; no voters:
I20240502 14:24:06.662009 7633 raft_consensus.cc:2454] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 333acff728274715a9816c286921ddd0 in term 3.
I20240502 14:24:06.662231 9719 raft_consensus.cc:2781] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:24:06.662619 9719 raft_consensus.cc:686] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 3 LEADER]: Becoming Leader. State: Replica: 333acff728274715a9816c286921ddd0, State: Running, Role: LEADER
I20240502 14:24:06.662969 9719 consensus_queue.cc:237] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } }
I20240502 14:24:06.665525 5591 catalog_manager.cc:5484] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 reported cstate change: term changed from 2 to 3, leader changed from 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66) to 333acff728274715a9816c286921ddd0 (127.5.107.68). New cstate: current_term: 3 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:24:07.056828 7633 raft_consensus.cc:1261] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e [term 3 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 14:24:07.056828 7504 raft_consensus.cc:1261] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 [term 3 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 14:24:07.057363 9719 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:24:07.057631 9815 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:24:07.062291 5864 raft_consensus.cc:1261] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 3 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 14:24:07.062916 9830 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:24:07.064880 6122 consensus_queue.cc:237] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } }
I20240502 14:24:07.066963 7633 raft_consensus.cc:1261] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e [term 3 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 14:24:07.068806 9830 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:24:07.071115 9815 raft_consensus.cc:2932] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } }
I20240502 14:24:07.071722 7633 raft_consensus.cc:2932] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } }
I20240502 14:24:07.072235 7503 raft_consensus.cc:1261] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 [term 3 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 14:24:07.073626 9719 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:24:07.075202 5578 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5358f204e4324db78adad5af90c17b86 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:24:07.074877 7503 raft_consensus.cc:2932] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } }
I20240502 14:24:07.076959 5592 catalog_manager.cc:5484] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e reported cstate change: config changed from index 9 to 11, VOTER 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66) evicted. New cstate: current_term: 3 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } }
I20240502 14:24:07.080765 5844 tablet_service.cc:1514] Processing DeleteTablet for tablet 5358f204e4324db78adad5af90c17b86 with delete_type TABLET_DATA_TOMBSTONED (TS 83e62d4e5f8a4316bc1bcb8c18458081 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:48002
I20240502 14:24:07.081657 9834 tablet_replica.cc:330] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081: stopping tablet replica
I20240502 14:24:07.081881 9834 raft_consensus.cc:2229] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 3 FOLLOWER]: Raft consensus shutting down.
W20240502 14:24:07.082077 6057 consensus_peers.cc:485] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:24:07.082144 9834 raft_consensus.cc:2258] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:24:07.082983 9834 ts_tablet_manager.cc:1875] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:24:07.086673 9834 ts_tablet_manager.cc:1888] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.10
I20240502 14:24:07.086771 9834 log.cc:1196] T 5358f204e4324db78adad5af90c17b86 P 83e62d4e5f8a4316bc1bcb8c18458081: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/wals/5358f204e4324db78adad5af90c17b86
I20240502 14:24:07.087365 5577 catalog_manager.cc:4830] TS 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015): tablet 5358f204e4324db78adad5af90c17b86 (table rebalance_test_table_3 [id=5a9100430c684a96b7fd7563ec356139]) successfully deleted
I20240502 14:24:07.139672 6122 consensus_queue.cc:237] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: NON_VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: true } }
I20240502 14:24:07.142004 7633 raft_consensus.cc:1261] T 08d82302438544e4925a4f511f13fb7e P 83b0e3c0ba724317832f8505a1ffb66e [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:24:07.142189 5993 raft_consensus.cc:1261] T 08d82302438544e4925a4f511f13fb7e P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 14:24:07.142432 6060 consensus_peers.cc:485] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 -> Peer 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71:42329): Couldn't send request to peer 681fd3c49e914094bf28a7df155cffc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08d82302438544e4925a4f511f13fb7e. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:24:07.142606 9830 consensus_queue.cc:1035] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:24:07.142885 9815 consensus_queue.cc:1035] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:24:07.145483 9719 raft_consensus.cc:2932] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: NON_VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: true } } }
I20240502 14:24:07.146085 5993 raft_consensus.cc:2932] T 08d82302438544e4925a4f511f13fb7e P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: NON_VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: true } } }
I20240502 14:24:07.146137 7633 raft_consensus.cc:2932] T 08d82302438544e4925a4f511f13fb7e P 83b0e3c0ba724317832f8505a1ffb66e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: NON_VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: true } } }
I20240502 14:24:07.149928 5591 catalog_manager.cc:5484] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 reported cstate change: config changed from index 6 to 7, NON_VOTER 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) added. New cstate: current_term: 1 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: NON_VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:24:07.167163 5864 consensus_queue.cc:237] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } }
I20240502 14:24:07.169466 7242 raft_consensus.cc:1261] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2 [term 1 FOLLOWER]: Refusing update from remote peer 83e62d4e5f8a4316bc1bcb8c18458081: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:24:07.175426 9781 consensus_queue.cc:1035] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:24:07.176535 6122 raft_consensus.cc:1261] T 81cfc8c8608b4cb0b7c303a910791d4b P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Refusing update from remote peer 83e62d4e5f8a4316bc1bcb8c18458081: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:24:07.177040 9855 consensus_queue.cc:1035] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:24:07.181571 9781 raft_consensus.cc:2932] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } } }
I20240502 14:24:07.186889 7242 raft_consensus.cc:2932] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } } }
I20240502 14:24:07.190295 5592 catalog_manager.cc:5484] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 reported cstate change: config changed from index 6 to 7, NON_VOTER 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) added. New cstate: current_term: 1 leader_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:24:07.193603 6123 raft_consensus.cc:2932] T 81cfc8c8608b4cb0b7c303a910791d4b P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } } }
W20240502 14:24:07.197521 5799 consensus_peers.cc:485] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70:40027): Couldn't send request to peer 806206bb049040bf8db48a7d8d3840c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81cfc8c8608b4cb0b7c303a910791d4b. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:24:07.284703 5927 consensus_peers.cc:485] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:24:07.364244 5576 catalog_manager.cc:4573] TS 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a6647f23da6c4430a9f84e378dd345a4: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
I20240502 14:24:07.601125 9886 ts_tablet_manager.cc:902] T 81cfc8c8608b4cb0b7c303a910791d4b P 806206bb049040bf8db48a7d8d3840c1: Initiating tablet copy from peer 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015)
I20240502 14:24:07.602512 9886 tablet_copy_client.cc:308] T 81cfc8c8608b4cb0b7c303a910791d4b P 806206bb049040bf8db48a7d8d3840c1: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:34015
I20240502 14:24:07.608817 5884 tablet_copy_service.cc:140] P 83e62d4e5f8a4316bc1bcb8c18458081: Received BeginTabletCopySession request for tablet 81cfc8c8608b4cb0b7c303a910791d4b from peer 806206bb049040bf8db48a7d8d3840c1 ({username='slave'} at 127.5.107.70:45027)
I20240502 14:24:07.608994 5884 tablet_copy_service.cc:161] P 83e62d4e5f8a4316bc1bcb8c18458081: Beginning new tablet copy session on tablet 81cfc8c8608b4cb0b7c303a910791d4b from peer 806206bb049040bf8db48a7d8d3840c1 at {username='slave'} at 127.5.107.70:45027: session id = 806206bb049040bf8db48a7d8d3840c1-81cfc8c8608b4cb0b7c303a910791d4b
I20240502 14:24:07.611953 5884 tablet_copy_source_session.cc:205] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:24:07.613207 9886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81cfc8c8608b4cb0b7c303a910791d4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:07.617069 9886 tablet_copy_client.cc:783] T 81cfc8c8608b4cb0b7c303a910791d4b P 806206bb049040bf8db48a7d8d3840c1: tablet copy: Starting download of 0 data blocks...
I20240502 14:24:07.617280 9886 tablet_copy_client.cc:647] T 81cfc8c8608b4cb0b7c303a910791d4b P 806206bb049040bf8db48a7d8d3840c1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:24:07.618595 9890 tablet_copy_client.cc:1002] T 81cfc8c8608b4cb0b7c303a910791d4b P 806206bb049040bf8db48a7d8d3840c1: tablet copy: Injecting latency into file download: 1500
I20240502 14:24:07.636857 9892 ts_tablet_manager.cc:902] T 08d82302438544e4925a4f511f13fb7e P 681fd3c49e914094bf28a7df155cffc9: Initiating tablet copy from peer 333acff728274715a9816c286921ddd0 (127.5.107.68:45913)
I20240502 14:24:07.638059 9892 tablet_copy_client.cc:308] T 08d82302438544e4925a4f511f13fb7e P 681fd3c49e914094bf28a7df155cffc9: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.68:45913
I20240502 14:24:07.638631 6144 tablet_copy_service.cc:140] P 333acff728274715a9816c286921ddd0: Received BeginTabletCopySession request for tablet 08d82302438544e4925a4f511f13fb7e from peer 681fd3c49e914094bf28a7df155cffc9 ({username='slave'} at 127.5.107.71:48765)
I20240502 14:24:07.638772 6144 tablet_copy_service.cc:161] P 333acff728274715a9816c286921ddd0: Beginning new tablet copy session on tablet 08d82302438544e4925a4f511f13fb7e from peer 681fd3c49e914094bf28a7df155cffc9 at {username='slave'} at 127.5.107.71:48765: session id = 681fd3c49e914094bf28a7df155cffc9-08d82302438544e4925a4f511f13fb7e
I20240502 14:24:07.640913 6144 tablet_copy_source_session.cc:205] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:24:07.642084 9892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08d82302438544e4925a4f511f13fb7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:07.646550 9892 tablet_copy_client.cc:783] T 08d82302438544e4925a4f511f13fb7e P 681fd3c49e914094bf28a7df155cffc9: tablet copy: Starting download of 0 data blocks...
I20240502 14:24:07.646766 9892 tablet_copy_client.cc:647] T 08d82302438544e4925a4f511f13fb7e P 681fd3c49e914094bf28a7df155cffc9: tablet copy: Starting download of 1 WAL segments...
I20240502 14:24:07.648159 9896 tablet_copy_client.cc:1002] T 08d82302438544e4925a4f511f13fb7e P 681fd3c49e914094bf28a7df155cffc9: tablet copy: Injecting latency into file download: 1500
W20240502 14:24:07.875404 6057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:24:08.264755 6057 consensus_peers.cc:485] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:24:08.421976 5797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:24:08.535617 5927 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:24:08.536444 5927 consensus_peers.cc:485] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:24:08.619153 6057 consensus_peers.cc:485] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:24:08.726071 5797 consensus_peers.cc:485] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:24:08.936640 5797 consensus_peers.cc:485] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:24:09.122555 9886 tablet_copy_client.cc:523] T 81cfc8c8608b4cb0b7c303a910791d4b P 806206bb049040bf8db48a7d8d3840c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:24:09.124805 9886 tablet_bootstrap.cc:492] T 81cfc8c8608b4cb0b7c303a910791d4b P 806206bb049040bf8db48a7d8d3840c1: Bootstrap starting.
I20240502 14:24:09.148582 9892 tablet_copy_client.cc:523] T 08d82302438544e4925a4f511f13fb7e P 681fd3c49e914094bf28a7df155cffc9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:24:09.151713 9892 tablet_bootstrap.cc:492] T 08d82302438544e4925a4f511f13fb7e P 681fd3c49e914094bf28a7df155cffc9: Bootstrap starting.
I20240502 14:24:09.153122 9886 tablet_bootstrap.cc:492] T 81cfc8c8608b4cb0b7c303a910791d4b P 806206bb049040bf8db48a7d8d3840c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:24:09.153519 9886 tablet_bootstrap.cc:492] T 81cfc8c8608b4cb0b7c303a910791d4b P 806206bb049040bf8db48a7d8d3840c1: Bootstrap complete.
I20240502 14:24:09.153757 9886 ts_tablet_manager.cc:1367] T 81cfc8c8608b4cb0b7c303a910791d4b P 806206bb049040bf8db48a7d8d3840c1: Time spent bootstrapping tablet: real 0.029s user 0.023s sys 0.007s
I20240502 14:24:09.154356 9886 raft_consensus.cc:348] T 81cfc8c8608b4cb0b7c303a910791d4b P 806206bb049040bf8db48a7d8d3840c1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } }
I20240502 14:24:09.154664 9886 raft_consensus.cc:729] T 81cfc8c8608b4cb0b7c303a910791d4b P 806206bb049040bf8db48a7d8d3840c1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 806206bb049040bf8db48a7d8d3840c1, State: Initialized, Role: LEARNER
I20240502 14:24:09.154912 9886 consensus_queue.cc:260] T 81cfc8c8608b4cb0b7c303a910791d4b P 806206bb049040bf8db48a7d8d3840c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } }
I20240502 14:24:09.155401 9886 ts_tablet_manager.cc:1398] T 81cfc8c8608b4cb0b7c303a910791d4b P 806206bb049040bf8db48a7d8d3840c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:24:09.156178 5884 tablet_copy_service.cc:342] P 83e62d4e5f8a4316bc1bcb8c18458081: Request end of tablet copy session 806206bb049040bf8db48a7d8d3840c1-81cfc8c8608b4cb0b7c303a910791d4b received from {username='slave'} at 127.5.107.70:45027
I20240502 14:24:09.156333 5884 tablet_copy_service.cc:433] P 83e62d4e5f8a4316bc1bcb8c18458081: ending tablet copy session 806206bb049040bf8db48a7d8d3840c1-81cfc8c8608b4cb0b7c303a910791d4b on tablet 81cfc8c8608b4cb0b7c303a910791d4b with peer 806206bb049040bf8db48a7d8d3840c1
I20240502 14:24:09.191933 9892 tablet_bootstrap.cc:492] T 08d82302438544e4925a4f511f13fb7e P 681fd3c49e914094bf28a7df155cffc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:24:09.192348 9892 tablet_bootstrap.cc:492] T 08d82302438544e4925a4f511f13fb7e P 681fd3c49e914094bf28a7df155cffc9: Bootstrap complete.
I20240502 14:24:09.192592 9892 ts_tablet_manager.cc:1367] T 08d82302438544e4925a4f511f13fb7e P 681fd3c49e914094bf28a7df155cffc9: Time spent bootstrapping tablet: real 0.041s user 0.030s sys 0.008s
I20240502 14:24:09.193296 9892 raft_consensus.cc:348] T 08d82302438544e4925a4f511f13fb7e P 681fd3c49e914094bf28a7df155cffc9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: NON_VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: true } }
I20240502 14:24:09.193533 9892 raft_consensus.cc:729] T 08d82302438544e4925a4f511f13fb7e P 681fd3c49e914094bf28a7df155cffc9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 681fd3c49e914094bf28a7df155cffc9, State: Initialized, Role: LEARNER
I20240502 14:24:09.193730 9892 consensus_queue.cc:260] T 08d82302438544e4925a4f511f13fb7e P 681fd3c49e914094bf28a7df155cffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: NON_VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: true } }
I20240502 14:24:09.194353 9892 ts_tablet_manager.cc:1398] T 08d82302438544e4925a4f511f13fb7e P 681fd3c49e914094bf28a7df155cffc9: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240502 14:24:09.195109 6144 tablet_copy_service.cc:342] P 333acff728274715a9816c286921ddd0: Request end of tablet copy session 681fd3c49e914094bf28a7df155cffc9-08d82302438544e4925a4f511f13fb7e received from {username='slave'} at 127.5.107.71:48765
I20240502 14:24:09.195222 6144 tablet_copy_service.cc:433] P 333acff728274715a9816c286921ddd0: ending tablet copy session 681fd3c49e914094bf28a7df155cffc9-08d82302438544e4925a4f511f13fb7e on tablet 08d82302438544e4925a4f511f13fb7e with peer 681fd3c49e914094bf28a7df155cffc9
W20240502 14:24:09.484473 6057 consensus_peers.cc:485] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:24:09.531023 7503 raft_consensus.cc:1203] T 08d82302438544e4925a4f511f13fb7e P 681fd3c49e914094bf28a7df155cffc9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20240502 14:24:09.614420 5927 consensus_peers.cc:485] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:24:09.620824 7373 raft_consensus.cc:1203] T 81cfc8c8608b4cb0b7c303a910791d4b P 806206bb049040bf8db48a7d8d3840c1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:24:09.960489 9930 raft_consensus.cc:1050] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0: attempting to promote NON_VOTER 681fd3c49e914094bf28a7df155cffc9 to VOTER
I20240502 14:24:09.961225 9930 consensus_queue.cc:237] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } }
I20240502 14:24:09.963320 7503 raft_consensus.cc:1261] T 08d82302438544e4925a4f511f13fb7e P 681fd3c49e914094bf28a7df155cffc9 [term 1 LEARNER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:24:09.963868 5993 raft_consensus.cc:1261] T 08d82302438544e4925a4f511f13fb7e P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:24:09.963815 10016 consensus_queue.cc:1035] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:24:09.964033 7633 raft_consensus.cc:1261] T 08d82302438544e4925a4f511f13fb7e P 83b0e3c0ba724317832f8505a1ffb66e [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:24:09.964816 10014 consensus_queue.cc:1035] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:24:09.965282 10016 consensus_queue.cc:1035] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:24:09.968320 10016 raft_consensus.cc:2932] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } }
I20240502 14:24:09.971480 7633 raft_consensus.cc:2932] T 08d82302438544e4925a4f511f13fb7e P 83b0e3c0ba724317832f8505a1ffb66e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } }
I20240502 14:24:09.975159 7503 raft_consensus.cc:2932] T 08d82302438544e4925a4f511f13fb7e P 681fd3c49e914094bf28a7df155cffc9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } }
I20240502 14:24:09.975453 5993 raft_consensus.cc:2932] T 08d82302438544e4925a4f511f13fb7e P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } }
I20240502 14:24:09.976083 5592 catalog_manager.cc:5484] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 reported cstate change: config changed from index 7 to 8, 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:24:09.984509 6122 consensus_queue.cc:237] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } }
I20240502 14:24:09.985846 7504 raft_consensus.cc:1261] T 08d82302438544e4925a4f511f13fb7e P 681fd3c49e914094bf28a7df155cffc9 [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:24:09.985836 7633 raft_consensus.cc:1261] T 08d82302438544e4925a4f511f13fb7e P 83b0e3c0ba724317832f8505a1ffb66e [term 1 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:24:09.986292 10017 consensus_queue.cc:1035] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:24:09.986542 9930 consensus_queue.cc:1035] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:24:09.988365 10014 raft_consensus.cc:2932] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 181b466de0084fcbace0ca327fb1cbef (127.5.107.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } }
I20240502 14:24:09.988880 7504 raft_consensus.cc:2932] T 08d82302438544e4925a4f511f13fb7e P 681fd3c49e914094bf28a7df155cffc9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 181b466de0084fcbace0ca327fb1cbef (127.5.107.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } }
I20240502 14:24:09.991320 5578 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 08d82302438544e4925a4f511f13fb7e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:24:09.992074 7633 raft_consensus.cc:2932] T 08d82302438544e4925a4f511f13fb7e P 83b0e3c0ba724317832f8505a1ffb66e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 181b466de0084fcbace0ca327fb1cbef (127.5.107.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } }
I20240502 14:24:09.992308 5583 catalog_manager.cc:5484] T 08d82302438544e4925a4f511f13fb7e P 333acff728274715a9816c286921ddd0 reported cstate change: config changed from index 8 to 9, VOTER 181b466de0084fcbace0ca327fb1cbef (127.5.107.67) evicted. New cstate: current_term: 1 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:24:09.995514 5973 tablet_service.cc:1514] Processing DeleteTablet for tablet 08d82302438544e4925a4f511f13fb7e with delete_type TABLET_DATA_TOMBSTONED (TS 181b466de0084fcbace0ca327fb1cbef not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:36800
I20240502 14:24:09.996102 10026 tablet_replica.cc:330] T 08d82302438544e4925a4f511f13fb7e P 181b466de0084fcbace0ca327fb1cbef: stopping tablet replica
I20240502 14:24:09.996299 10026 raft_consensus.cc:2229] T 08d82302438544e4925a4f511f13fb7e P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:24:09.996565 10026 raft_consensus.cc:2258] T 08d82302438544e4925a4f511f13fb7e P 181b466de0084fcbace0ca327fb1cbef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:24:09.997301 10026 ts_tablet_manager.cc:1875] T 08d82302438544e4925a4f511f13fb7e P 181b466de0084fcbace0ca327fb1cbef: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:24:10.001057 10026 ts_tablet_manager.cc:1888] T 08d82302438544e4925a4f511f13fb7e P 181b466de0084fcbace0ca327fb1cbef: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 14:24:10.001168 10026 log.cc:1196] T 08d82302438544e4925a4f511f13fb7e P 181b466de0084fcbace0ca327fb1cbef: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal/wals/08d82302438544e4925a4f511f13fb7e
I20240502 14:24:10.001858 5578 catalog_manager.cc:4830] TS 181b466de0084fcbace0ca327fb1cbef (127.5.107.67:41517): tablet 08d82302438544e4925a4f511f13fb7e (table rebalance_test_table_1 [id=a3f5812bc1aa47c1990b9a769141e329]) successfully deleted
I20240502 14:24:10.191802 10042 raft_consensus.cc:1050] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081: attempting to promote NON_VOTER 806206bb049040bf8db48a7d8d3840c1 to VOTER
I20240502 14:24:10.192543 10042 consensus_queue.cc:237] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } }
I20240502 14:24:10.194662 7373 raft_consensus.cc:1261] T 81cfc8c8608b4cb0b7c303a910791d4b P 806206bb049040bf8db48a7d8d3840c1 [term 1 LEARNER]: Refusing update from remote peer 83e62d4e5f8a4316bc1bcb8c18458081: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:24:10.195065 6122 raft_consensus.cc:1261] T 81cfc8c8608b4cb0b7c303a910791d4b P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Refusing update from remote peer 83e62d4e5f8a4316bc1bcb8c18458081: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:24:10.195273 7242 raft_consensus.cc:1261] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2 [term 1 FOLLOWER]: Refusing update from remote peer 83e62d4e5f8a4316bc1bcb8c18458081: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:24:10.195333 10043 consensus_queue.cc:1035] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:24:10.195796 10045 consensus_queue.cc:1035] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:24:10.196202 10043 consensus_queue.cc:1035] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:24:10.199082 9928 raft_consensus.cc:2932] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:24:10.199558 7373 raft_consensus.cc:2932] T 81cfc8c8608b4cb0b7c303a910791d4b P 806206bb049040bf8db48a7d8d3840c1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:24:10.203998 7242 raft_consensus.cc:2932] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:24:10.205664 5591 catalog_manager.cc:5484] T 81cfc8c8608b4cb0b7c303a910791d4b P 806206bb049040bf8db48a7d8d3840c1 reported cstate change: config changed from index 7 to 8, 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:24:10.208590 6122 raft_consensus.cc:2932] T 81cfc8c8608b4cb0b7c303a910791d4b P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:24:10.281704 5864 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "81cfc8c8608b4cb0b7c303a910791d4b"
dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41710
I20240502 14:24:10.281900 5864 raft_consensus.cc:595] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:24:10.498288 5864 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "81cfc8c8608b4cb0b7c303a910791d4b"
dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41728
I20240502 14:24:10.498467 5864 raft_consensus.cc:595] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:24:10.589098 10049 raft_consensus.cc:979] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081: : Instructing follower 3416b273466c495298342b6e892015c2 to start an election
I20240502 14:24:10.589298 10049 raft_consensus.cc:1067] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 LEADER]: Signalling peer 3416b273466c495298342b6e892015c2 to start an election
I20240502 14:24:10.590216 7242 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "81cfc8c8608b4cb0b7c303a910791d4b"
dest_uuid: "3416b273466c495298342b6e892015c2"
from {username='slave'} at 127.5.107.66:59011
I20240502 14:24:10.590556 7242 raft_consensus.cc:482] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:24:10.590672 7242 raft_consensus.cc:3037] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:24:10.592665 7242 raft_consensus.cc:504] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } }
I20240502 14:24:10.594256 7242 leader_election.cc:288] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2 [CANDIDATE]: Term 2 election: Requested vote from peers 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015), 333acff728274715a9816c286921ddd0 (127.5.107.68:45913), 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70:40027)
I20240502 14:24:10.594647 5864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81cfc8c8608b4cb0b7c303a910791d4b" candidate_uuid: "3416b273466c495298342b6e892015c2" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "83e62d4e5f8a4316bc1bcb8c18458081"
I20240502 14:24:10.594870 5864 raft_consensus.cc:3032] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:24:10.594934 5864 raft_consensus.cc:729] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83e62d4e5f8a4316bc1bcb8c18458081, State: Running, Role: LEADER
I20240502 14:24:10.595176 5864 consensus_queue.cc:260] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } }
I20240502 14:24:10.595499 5864 raft_consensus.cc:3037] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:24:10.597539 6122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81cfc8c8608b4cb0b7c303a910791d4b" candidate_uuid: "3416b273466c495298342b6e892015c2" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "333acff728274715a9816c286921ddd0"
I20240502 14:24:10.597738 6122 raft_consensus.cc:3037] T 81cfc8c8608b4cb0b7c303a910791d4b P 333acff728274715a9816c286921ddd0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:24:10.599859 5864 raft_consensus.cc:2454] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3416b273466c495298342b6e892015c2 in term 2.
I20240502 14:24:10.600325 6122 raft_consensus.cc:2454] T 81cfc8c8608b4cb0b7c303a910791d4b P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3416b273466c495298342b6e892015c2 in term 2.
I20240502 14:24:10.600796 7177 leader_election.cc:302] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 333acff728274715a9816c286921ddd0, 3416b273466c495298342b6e892015c2, 83e62d4e5f8a4316bc1bcb8c18458081; no voters:
I20240502 14:24:10.601245 10050 raft_consensus.cc:2781] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:24:10.602280 10050 raft_consensus.cc:686] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2 [term 2 LEADER]: Becoming Leader. State: Replica: 3416b273466c495298342b6e892015c2, State: Running, Role: LEADER
I20240502 14:24:10.602784 10050 consensus_queue.cc:237] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } }
I20240502 14:24:10.603130 7373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81cfc8c8608b4cb0b7c303a910791d4b" candidate_uuid: "3416b273466c495298342b6e892015c2" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "806206bb049040bf8db48a7d8d3840c1"
I20240502 14:24:10.603377 7373 raft_consensus.cc:3037] T 81cfc8c8608b4cb0b7c303a910791d4b P 806206bb049040bf8db48a7d8d3840c1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:24:10.605890 7373 raft_consensus.cc:2454] T 81cfc8c8608b4cb0b7c303a910791d4b P 806206bb049040bf8db48a7d8d3840c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3416b273466c495298342b6e892015c2 in term 2.
I20240502 14:24:10.606434 5591 catalog_manager.cc:5484] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2 reported cstate change: term changed from 1 to 2, leader changed from 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66) to 3416b273466c495298342b6e892015c2 (127.5.107.69). New cstate: current_term: 2 leader_uuid: "3416b273466c495298342b6e892015c2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 14:24:10.846967 5576 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240502 14:24:10.848085 5576 catalog_manager.cc:4573] TS 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7d944a63fdf24ecd8fadbad3b620af0a: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
W20240502 14:24:11.011907 5927 consensus_peers.cc:485] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:24:11.017158 6122 raft_consensus.cc:1261] T 81cfc8c8608b4cb0b7c303a910791d4b P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Refusing update from remote peer 3416b273466c495298342b6e892015c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:24:11.017230 5864 raft_consensus.cc:1261] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Refusing update from remote peer 3416b273466c495298342b6e892015c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:24:11.017807 10050 consensus_queue.cc:1035] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:24:11.018177 10073 consensus_queue.cc:1035] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:24:11.026633 7373 raft_consensus.cc:1261] T 81cfc8c8608b4cb0b7c303a910791d4b P 806206bb049040bf8db48a7d8d3840c1 [term 2 FOLLOWER]: Refusing update from remote peer 3416b273466c495298342b6e892015c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:24:11.027285 10073 consensus_queue.cc:1035] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 14:24:11.085124 6057 consensus_peers.cc:485] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:24:11.087332 5797 consensus_peers.cc:485] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:24:11.144945 6057 consensus_peers.cc:485] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:24:11.170852 5797 consensus_peers.cc:485] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:24:11.574452 7242 consensus_queue.cc:237] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } }
I20240502 14:24:11.576297 7373 raft_consensus.cc:1261] T 81cfc8c8608b4cb0b7c303a910791d4b P 806206bb049040bf8db48a7d8d3840c1 [term 2 FOLLOWER]: Refusing update from remote peer 3416b273466c495298342b6e892015c2: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:24:11.576361 6122 raft_consensus.cc:1261] T 81cfc8c8608b4cb0b7c303a910791d4b P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Refusing update from remote peer 3416b273466c495298342b6e892015c2: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:24:11.576823 10108 consensus_queue.cc:1035] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:24:11.577162 10109 consensus_queue.cc:1035] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:24:11.579442 10108 raft_consensus.cc:2932] T 81cfc8c8608b4cb0b7c303a910791d4b P 3416b273466c495298342b6e892015c2 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:24:11.579932 7373 raft_consensus.cc:2932] T 81cfc8c8608b4cb0b7c303a910791d4b P 806206bb049040bf8db48a7d8d3840c1 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:24:11.579938 6122 raft_consensus.cc:2932] T 81cfc8c8608b4cb0b7c303a910791d4b P 333acff728274715a9816c286921ddd0 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:24:11.583539 5578 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 81cfc8c8608b4cb0b7c303a910791d4b with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:24:11.585803 5591 catalog_manager.cc:5484] T 81cfc8c8608b4cb0b7c303a910791d4b P 333acff728274715a9816c286921ddd0 reported cstate change: config changed from index 8 to 10, VOTER 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66) evicted. New cstate: current_term: 2 leader_uuid: "3416b273466c495298342b6e892015c2" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "3416b273466c495298342b6e892015c2" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 37277 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:24:11.590023 5844 tablet_service.cc:1514] Processing DeleteTablet for tablet 81cfc8c8608b4cb0b7c303a910791d4b with delete_type TABLET_DATA_TOMBSTONED (TS 83e62d4e5f8a4316bc1bcb8c18458081 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:48002
I20240502 14:24:11.590655 10114 tablet_replica.cc:330] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081: stopping tablet replica
I20240502 14:24:11.590885 10114 raft_consensus.cc:2229] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:24:11.591120 10114 raft_consensus.cc:2258] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:24:11.591892 10114 ts_tablet_manager.cc:1875] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:24:11.596012 10114 ts_tablet_manager.cc:1888] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 14:24:11.596128 10114 log.cc:1196] T 81cfc8c8608b4cb0b7c303a910791d4b P 83e62d4e5f8a4316bc1bcb8c18458081: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/wals/81cfc8c8608b4cb0b7c303a910791d4b
I20240502 14:24:11.596773 5577 catalog_manager.cc:4830] TS 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015): tablet 81cfc8c8608b4cb0b7c303a910791d4b (table rebalance_test_table_1 [id=a3f5812bc1aa47c1990b9a769141e329]) successfully deleted
I20240502 14:24:11.769637 6122 consensus_queue.cc:237] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } }
I20240502 14:24:11.772686 7373 raft_consensus.cc:1261] T a6647f23da6c4430a9f84e378dd345a4 P 806206bb049040bf8db48a7d8d3840c1 [term 2 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
W20240502 14:24:11.773253 6057 consensus_peers.cc:485] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 -> Peer 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72:35323): Couldn't send request to peer 83b0e3c0ba724317832f8505a1ffb66e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6647f23da6c4430a9f84e378dd345a4. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:24:11.773270 10017 consensus_queue.cc:1035] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:24:11.774561 5864 raft_consensus.cc:1261] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:24:11.775125 10127 consensus_queue.cc:1035] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:24:11.775686 10127 raft_consensus.cc:2932] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } } }
I20240502 14:24:11.776225 7373 raft_consensus.cc:2932] T a6647f23da6c4430a9f84e378dd345a4 P 806206bb049040bf8db48a7d8d3840c1 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } } }
I20240502 14:24:11.779510 5586 catalog_manager.cc:5484] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 reported cstate change: config changed from index 8 to 9, NON_VOTER 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) added. New cstate: current_term: 2 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:24:11.789824 5864 raft_consensus.cc:2932] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } } }
W20240502 14:24:12.090623 6057 consensus_peers.cc:485] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:24:12.205171 5927 consensus_peers.cc:485] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:24:12.228691 10146 ts_tablet_manager.cc:902] T a6647f23da6c4430a9f84e378dd345a4 P 83b0e3c0ba724317832f8505a1ffb66e: Initiating tablet copy from peer 333acff728274715a9816c286921ddd0 (127.5.107.68:45913)
I20240502 14:24:12.229970 10146 tablet_copy_client.cc:308] T a6647f23da6c4430a9f84e378dd345a4 P 83b0e3c0ba724317832f8505a1ffb66e: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.68:45913
I20240502 14:24:12.230648 6144 tablet_copy_service.cc:140] P 333acff728274715a9816c286921ddd0: Received BeginTabletCopySession request for tablet a6647f23da6c4430a9f84e378dd345a4 from peer 83b0e3c0ba724317832f8505a1ffb66e ({username='slave'} at 127.5.107.72:46611)
I20240502 14:24:12.230780 6144 tablet_copy_service.cc:161] P 333acff728274715a9816c286921ddd0: Beginning new tablet copy session on tablet a6647f23da6c4430a9f84e378dd345a4 from peer 83b0e3c0ba724317832f8505a1ffb66e at {username='slave'} at 127.5.107.72:46611: session id = 83b0e3c0ba724317832f8505a1ffb66e-a6647f23da6c4430a9f84e378dd345a4
I20240502 14:24:12.232829 6144 tablet_copy_source_session.cc:205] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:24:12.234035 10146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6647f23da6c4430a9f84e378dd345a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:12.237919 10146 tablet_copy_client.cc:783] T a6647f23da6c4430a9f84e378dd345a4 P 83b0e3c0ba724317832f8505a1ffb66e: tablet copy: Starting download of 0 data blocks...
I20240502 14:24:12.238137 10146 tablet_copy_client.cc:647] T a6647f23da6c4430a9f84e378dd345a4 P 83b0e3c0ba724317832f8505a1ffb66e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:24:12.239537 10150 tablet_copy_client.cc:1002] T a6647f23da6c4430a9f84e378dd345a4 P 83b0e3c0ba724317832f8505a1ffb66e: tablet copy: Injecting latency into file download: 1500
W20240502 14:24:13.115284 6057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:24:13.502827 5927 consensus_peers.cc:485] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:24:13.606539 5797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:24:13.607405 5797 consensus_peers.cc:485] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:24:13.610680 5927 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:24:13.654253 5797 consensus_peers.cc:485] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:24:13.739876 10146 tablet_copy_client.cc:523] T a6647f23da6c4430a9f84e378dd345a4 P 83b0e3c0ba724317832f8505a1ffb66e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:24:13.742415 10146 tablet_bootstrap.cc:492] T a6647f23da6c4430a9f84e378dd345a4 P 83b0e3c0ba724317832f8505a1ffb66e: Bootstrap starting.
W20240502 14:24:13.765607 6057 consensus_peers.cc:485] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:24:13.770377 10146 tablet_bootstrap.cc:492] T a6647f23da6c4430a9f84e378dd345a4 P 83b0e3c0ba724317832f8505a1ffb66e: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:24:13.770749 10146 tablet_bootstrap.cc:492] T a6647f23da6c4430a9f84e378dd345a4 P 83b0e3c0ba724317832f8505a1ffb66e: Bootstrap complete.
I20240502 14:24:13.770975 10146 ts_tablet_manager.cc:1367] T a6647f23da6c4430a9f84e378dd345a4 P 83b0e3c0ba724317832f8505a1ffb66e: Time spent bootstrapping tablet: real 0.029s user 0.022s sys 0.006s
I20240502 14:24:13.771550 10146 raft_consensus.cc:348] T a6647f23da6c4430a9f84e378dd345a4 P 83b0e3c0ba724317832f8505a1ffb66e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } }
I20240502 14:24:13.771752 10146 raft_consensus.cc:729] T a6647f23da6c4430a9f84e378dd345a4 P 83b0e3c0ba724317832f8505a1ffb66e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 83b0e3c0ba724317832f8505a1ffb66e, State: Initialized, Role: LEARNER
I20240502 14:24:13.771920 10146 consensus_queue.cc:260] T a6647f23da6c4430a9f84e378dd345a4 P 83b0e3c0ba724317832f8505a1ffb66e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: NON_VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: true } }
I20240502 14:24:13.772508 10146 ts_tablet_manager.cc:1398] T a6647f23da6c4430a9f84e378dd345a4 P 83b0e3c0ba724317832f8505a1ffb66e: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20240502 14:24:13.773216 6144 tablet_copy_service.cc:342] P 333acff728274715a9816c286921ddd0: Request end of tablet copy session 83b0e3c0ba724317832f8505a1ffb66e-a6647f23da6c4430a9f84e378dd345a4 received from {username='slave'} at 127.5.107.72:46611
I20240502 14:24:13.773356 6144 tablet_copy_service.cc:433] P 333acff728274715a9816c286921ddd0: ending tablet copy session 83b0e3c0ba724317832f8505a1ffb66e-a6647f23da6c4430a9f84e378dd345a4 on tablet a6647f23da6c4430a9f84e378dd345a4 with peer 83b0e3c0ba724317832f8505a1ffb66e
W20240502 14:24:13.851233 6057 consensus_peers.cc:485] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:24:13.893800 5576 catalog_manager.cc:4573] TS 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 25f9ebd715dc4f96901b60796ea846ad: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
I20240502 14:24:14.239678 7633 raft_consensus.cc:1203] T a6647f23da6c4430a9f84e378dd345a4 P 83b0e3c0ba724317832f8505a1ffb66e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
W20240502 14:24:14.497098 6057 consensus_peers.cc:485] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:24:14.554687 5927 consensus_peers.cc:485] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:24:14.704710 10215 raft_consensus.cc:1050] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0: attempting to promote NON_VOTER 83b0e3c0ba724317832f8505a1ffb66e to VOTER
I20240502 14:24:14.705440 10215 consensus_queue.cc:237] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } }
I20240502 14:24:14.707646 5864 raft_consensus.cc:1261] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:24:14.707739 7633 raft_consensus.cc:1261] T a6647f23da6c4430a9f84e378dd345a4 P 83b0e3c0ba724317832f8505a1ffb66e [term 2 LEARNER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:24:14.707939 7373 raft_consensus.cc:1261] T a6647f23da6c4430a9f84e378dd345a4 P 806206bb049040bf8db48a7d8d3840c1 [term 2 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:24:14.708159 10217 consensus_queue.cc:1035] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:24:14.708571 10215 consensus_queue.cc:1035] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:24:14.708832 10209 consensus_queue.cc:1035] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:24:14.711477 10217 raft_consensus.cc:2932] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:24:14.712158 7633 raft_consensus.cc:2932] T a6647f23da6c4430a9f84e378dd345a4 P 83b0e3c0ba724317832f8505a1ffb66e [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:24:14.715397 5864 raft_consensus.cc:2932] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:24:14.716495 5586 catalog_manager.cc:5484] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 reported cstate change: config changed from index 9 to 10, 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:24:14.726840 7373 raft_consensus.cc:2932] T a6647f23da6c4430a9f84e378dd345a4 P 806206bb049040bf8db48a7d8d3840c1 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } attrs { replace: true } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:24:14.732980 6122 consensus_queue.cc:237] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } }
I20240502 14:24:14.734526 7373 raft_consensus.cc:1261] T a6647f23da6c4430a9f84e378dd345a4 P 806206bb049040bf8db48a7d8d3840c1 [term 2 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 14:24:14.734599 7633 raft_consensus.cc:1261] T a6647f23da6c4430a9f84e378dd345a4 P 83b0e3c0ba724317832f8505a1ffb66e [term 2 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 14:24:14.734941 10215 consensus_queue.cc:1035] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:24:14.735177 10209 consensus_queue.cc:1035] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:24:14.737149 10162 raft_consensus.cc:2932] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:24:14.737917 7633 raft_consensus.cc:2932] T a6647f23da6c4430a9f84e378dd345a4 P 83b0e3c0ba724317832f8505a1ffb66e [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:24:14.740530 5578 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a6647f23da6c4430a9f84e378dd345a4 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:24:14.740796 7373 raft_consensus.cc:2932] T a6647f23da6c4430a9f84e378dd345a4 P 806206bb049040bf8db48a7d8d3840c1 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } }
I20240502 14:24:14.742130 5583 catalog_manager.cc:5484] T a6647f23da6c4430a9f84e378dd345a4 P 333acff728274715a9816c286921ddd0 reported cstate change: config changed from index 10 to 11, VOTER 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66) evicted. New cstate: current_term: 2 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:24:14.745260 5844 tablet_service.cc:1514] Processing DeleteTablet for tablet a6647f23da6c4430a9f84e378dd345a4 with delete_type TABLET_DATA_TOMBSTONED (TS 83e62d4e5f8a4316bc1bcb8c18458081 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:48002
I20240502 14:24:14.745862 10230 tablet_replica.cc:330] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081: stopping tablet replica
I20240502 14:24:14.746006 10230 raft_consensus.cc:2229] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:24:14.746316 10230 raft_consensus.cc:2258] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:24:14.746927 10230 ts_tablet_manager.cc:1875] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:24:14.750620 10230 ts_tablet_manager.cc:1888] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240502 14:24:14.750730 10230 log.cc:1196] T a6647f23da6c4430a9f84e378dd345a4 P 83e62d4e5f8a4316bc1bcb8c18458081: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/wals/a6647f23da6c4430a9f84e378dd345a4
I20240502 14:24:14.751435 5577 catalog_manager.cc:4830] TS 83e62d4e5f8a4316bc1bcb8c18458081 (127.5.107.66:34015): tablet a6647f23da6c4430a9f84e378dd345a4 (table rebalance_test_table_2 [id=64688b21b92e4f8596609be0f50e11bf]) successfully deleted
I20240502 14:24:15.060323 5844 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:24:15.062844 5973 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:24:15.076118 6103 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:24:15.078300 7484 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:15.079169 7354 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:24:15.083276 7222 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:24:15.098349 7614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:15.174520 6122 consensus_queue.cc:237] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } }
I20240502 14:24:15.176586 7504 raft_consensus.cc:1261] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 [term 3 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240502 14:24:15.176820 7633 raft_consensus.cc:1261] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e [term 3 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240502 14:24:15.177055 10162 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:24:15.177390 10215 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240502 14:24:15.177604 6059 consensus_peers.cc:485] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 -> Peer 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70:40027): Couldn't send request to peer 806206bb049040bf8db48a7d8d3840c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5358f204e4324db78adad5af90c17b86. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:24:15.179847 10209 raft_consensus.cc:2932] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 11 to 12, NON_VOTER 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } } }
I20240502 14:24:15.180442 7633 raft_consensus.cc:2932] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 11 to 12, NON_VOTER 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } } }
I20240502 14:24:15.180548 7504 raft_consensus.cc:2932] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 11 to 12, NON_VOTER 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } } }
I20240502 14:24:15.185796 5586 catalog_manager.cc:5484] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 reported cstate change: config changed from index 11 to 12, NON_VOTER 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) added. New cstate: current_term: 3 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:24:15.579449 10294 ts_tablet_manager.cc:902] T 5358f204e4324db78adad5af90c17b86 P 806206bb049040bf8db48a7d8d3840c1: Initiating tablet copy from peer 333acff728274715a9816c286921ddd0 (127.5.107.68:45913)
I20240502 14:24:15.581041 10294 tablet_copy_client.cc:308] T 5358f204e4324db78adad5af90c17b86 P 806206bb049040bf8db48a7d8d3840c1: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.68:45913
I20240502 14:24:15.581681 6144 tablet_copy_service.cc:140] P 333acff728274715a9816c286921ddd0: Received BeginTabletCopySession request for tablet 5358f204e4324db78adad5af90c17b86 from peer 806206bb049040bf8db48a7d8d3840c1 ({username='slave'} at 127.5.107.70:36379)
I20240502 14:24:15.581862 6144 tablet_copy_service.cc:161] P 333acff728274715a9816c286921ddd0: Beginning new tablet copy session on tablet 5358f204e4324db78adad5af90c17b86 from peer 806206bb049040bf8db48a7d8d3840c1 at {username='slave'} at 127.5.107.70:36379: session id = 806206bb049040bf8db48a7d8d3840c1-5358f204e4324db78adad5af90c17b86
I20240502 14:24:15.584724 6144 tablet_copy_source_session.cc:205] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:24:15.585934 10294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5358f204e4324db78adad5af90c17b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:15.589932 10294 tablet_copy_client.cc:783] T 5358f204e4324db78adad5af90c17b86 P 806206bb049040bf8db48a7d8d3840c1: tablet copy: Starting download of 0 data blocks...
I20240502 14:24:15.590134 10294 tablet_copy_client.cc:647] T 5358f204e4324db78adad5af90c17b86 P 806206bb049040bf8db48a7d8d3840c1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:24:15.591459 10298 tablet_copy_client.cc:1002] T 5358f204e4324db78adad5af90c17b86 P 806206bb049040bf8db48a7d8d3840c1: tablet copy: Injecting latency into file download: 1500
W20240502 14:24:15.900291 5927 consensus_peers.cc:485] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:24:16.088752 5797 consensus_peers.cc:485] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:24:16.178417 6057 consensus_peers.cc:485] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:24:16.245942 5797 consensus_peers.cc:485] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:24:16.273286 6057 consensus_peers.cc:485] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:24:17.006954 5927 consensus_peers.cc:485] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:24:17.091805 10294 tablet_copy_client.cc:523] T 5358f204e4324db78adad5af90c17b86 P 806206bb049040bf8db48a7d8d3840c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:24:17.094697 10294 tablet_bootstrap.cc:492] T 5358f204e4324db78adad5af90c17b86 P 806206bb049040bf8db48a7d8d3840c1: Bootstrap starting.
I20240502 14:24:17.124507 10294 tablet_bootstrap.cc:492] T 5358f204e4324db78adad5af90c17b86 P 806206bb049040bf8db48a7d8d3840c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:24:17.124864 10294 tablet_bootstrap.cc:492] T 5358f204e4324db78adad5af90c17b86 P 806206bb049040bf8db48a7d8d3840c1: Bootstrap complete.
I20240502 14:24:17.125090 10294 ts_tablet_manager.cc:1367] T 5358f204e4324db78adad5af90c17b86 P 806206bb049040bf8db48a7d8d3840c1: Time spent bootstrapping tablet: real 0.030s user 0.028s sys 0.000s
I20240502 14:24:17.125669 10294 raft_consensus.cc:348] T 5358f204e4324db78adad5af90c17b86 P 806206bb049040bf8db48a7d8d3840c1 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } }
I20240502 14:24:17.125864 10294 raft_consensus.cc:729] T 5358f204e4324db78adad5af90c17b86 P 806206bb049040bf8db48a7d8d3840c1 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 806206bb049040bf8db48a7d8d3840c1, State: Initialized, Role: LEARNER
I20240502 14:24:17.126026 10294 consensus_queue.cc:260] T 5358f204e4324db78adad5af90c17b86 P 806206bb049040bf8db48a7d8d3840c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } }
I20240502 14:24:17.126504 10294 ts_tablet_manager.cc:1398] T 5358f204e4324db78adad5af90c17b86 P 806206bb049040bf8db48a7d8d3840c1: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 14:24:17.127215 6144 tablet_copy_service.cc:342] P 333acff728274715a9816c286921ddd0: Request end of tablet copy session 806206bb049040bf8db48a7d8d3840c1-5358f204e4324db78adad5af90c17b86 received from {username='slave'} at 127.5.107.70:36379
I20240502 14:24:17.127350 6144 tablet_copy_service.cc:433] P 333acff728274715a9816c286921ddd0: ending tablet copy session 806206bb049040bf8db48a7d8d3840c1-5358f204e4324db78adad5af90c17b86 on tablet 5358f204e4324db78adad5af90c17b86 with peer 806206bb049040bf8db48a7d8d3840c1
W20240502 14:24:17.204041 6057 consensus_peers.cc:485] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:24:17.330622 5576 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240502 14:24:17.331777 5576 catalog_manager.cc:4573] TS 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9fdda89a7743466a8aeceb543124fd79: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111)
I20240502 14:24:17.716287 7373 raft_consensus.cc:1203] T 5358f204e4324db78adad5af90c17b86 P 806206bb049040bf8db48a7d8d3840c1 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.11->[3.12-3.12] Dedup: 3.12->[]
W20240502 14:24:18.236465 6057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 14:24:18.326436 10369 raft_consensus.cc:1050] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0: attempting to promote NON_VOTER 806206bb049040bf8db48a7d8d3840c1 to VOTER
I20240502 14:24:18.327294 10369 consensus_queue.cc:237] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 9, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } }
I20240502 14:24:18.330665 7373 raft_consensus.cc:1261] T 5358f204e4324db78adad5af90c17b86 P 806206bb049040bf8db48a7d8d3840c1 [term 3 LEARNER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 14:24:18.331149 7633 raft_consensus.cc:1261] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e [term 3 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 14:24:18.331235 10370 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 14:24:18.331761 7504 raft_consensus.cc:1261] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 [term 3 FOLLOWER]: Refusing update from remote peer 333acff728274715a9816c286921ddd0: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 14:24:18.331966 10371 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 14:24:18.332376 10370 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 14:24:18.339504 10369 raft_consensus.cc:2932] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 12 to 13, 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:24:18.340152 7373 raft_consensus.cc:2932] T 5358f204e4324db78adad5af90c17b86 P 806206bb049040bf8db48a7d8d3840c1 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:24:18.340804 7503 raft_consensus.cc:2932] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:24:18.344537 7633 raft_consensus.cc:2932] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:24:18.345588 5583 catalog_manager.cc:5484] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 reported cstate change: config changed from index 12 to 13, 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "333acff728274715a9816c286921ddd0" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:24:18.357836 6122 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5358f204e4324db78adad5af90c17b86"
dest_uuid: "333acff728274715a9816c286921ddd0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44234
I20240502 14:24:18.357985 6122 raft_consensus.cc:595] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 3 LEADER]: Received request to transfer leadership
W20240502 14:24:18.415475 5927 consensus_peers.cc:485] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:24:18.575063 6122 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5358f204e4324db78adad5af90c17b86"
dest_uuid: "333acff728274715a9816c286921ddd0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44254
I20240502 14:24:18.575255 6122 raft_consensus.cc:595] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 3 LEADER]: Received request to transfer leadership
W20240502 14:24:18.644197 5797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:24:18.645023 5797 consensus_peers.cc:485] T 228133f033a44f59af1259cc1a1e9d55 P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:24:18.724916 10369 raft_consensus.cc:979] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0: : Instructing follower 681fd3c49e914094bf28a7df155cffc9 to start an election
I20240502 14:24:18.725087 10369 raft_consensus.cc:1067] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 3 LEADER]: Signalling peer 681fd3c49e914094bf28a7df155cffc9 to start an election
I20240502 14:24:18.725731 7503 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5358f204e4324db78adad5af90c17b86"
dest_uuid: "681fd3c49e914094bf28a7df155cffc9"
from {username='slave'} at 127.5.107.68:60595
I20240502 14:24:18.725977 7503 raft_consensus.cc:482] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:24:18.726091 7503 raft_consensus.cc:3037] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:24:18.727830 7503 raft_consensus.cc:504] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } }
I20240502 14:24:18.729220 7503 leader_election.cc:288] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 [CANDIDATE]: Term 4 election: Requested vote from peers 333acff728274715a9816c286921ddd0 (127.5.107.68:45913), 83b0e3c0ba724317832f8505a1ffb66e (127.5.107.72:35323), 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70:40027)
I20240502 14:24:18.729540 6122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5358f204e4324db78adad5af90c17b86" candidate_uuid: "681fd3c49e914094bf28a7df155cffc9" candidate_term: 4 candidate_status { last_received { term: 3 index: 13 } } ignore_live_leader: true dest_uuid: "333acff728274715a9816c286921ddd0"
I20240502 14:24:18.729728 6122 raft_consensus.cc:3032] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:24:18.729825 6122 raft_consensus.cc:729] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 333acff728274715a9816c286921ddd0, State: Running, Role: LEADER
I20240502 14:24:18.730072 6122 consensus_queue.cc:260] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 13, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } }
I20240502 14:24:18.730568 6122 raft_consensus.cc:3037] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:24:18.733261 6122 raft_consensus.cc:2454] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 681fd3c49e914094bf28a7df155cffc9 in term 4.
I20240502 14:24:18.736819 7373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5358f204e4324db78adad5af90c17b86" candidate_uuid: "681fd3c49e914094bf28a7df155cffc9" candidate_term: 4 candidate_status { last_received { term: 3 index: 13 } } ignore_live_leader: true dest_uuid: "806206bb049040bf8db48a7d8d3840c1"
I20240502 14:24:18.737071 7373 raft_consensus.cc:3037] T 5358f204e4324db78adad5af90c17b86 P 806206bb049040bf8db48a7d8d3840c1 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:24:18.737439 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5358f204e4324db78adad5af90c17b86" candidate_uuid: "681fd3c49e914094bf28a7df155cffc9" candidate_term: 4 candidate_status { last_received { term: 3 index: 13 } } ignore_live_leader: true dest_uuid: "83b0e3c0ba724317832f8505a1ffb66e"
I20240502 14:24:18.737601 7633 raft_consensus.cc:3037] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:24:18.738974 7373 raft_consensus.cc:2454] T 5358f204e4324db78adad5af90c17b86 P 806206bb049040bf8db48a7d8d3840c1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 681fd3c49e914094bf28a7df155cffc9 in term 4.
I20240502 14:24:18.739257 7633 raft_consensus.cc:2454] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 681fd3c49e914094bf28a7df155cffc9 in term 4.
I20240502 14:24:18.739367 7439 leader_election.cc:302] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 333acff728274715a9816c286921ddd0, 681fd3c49e914094bf28a7df155cffc9, 806206bb049040bf8db48a7d8d3840c1; no voters:
I20240502 14:24:18.739706 10381 raft_consensus.cc:2781] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:24:18.740085 10381 raft_consensus.cc:686] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 [term 4 LEADER]: Becoming Leader. State: Replica: 681fd3c49e914094bf28a7df155cffc9, State: Running, Role: LEADER
I20240502 14:24:18.740516 10381 consensus_queue.cc:237] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 13, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } }
I20240502 14:24:18.743633 5592 catalog_manager.cc:5484] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 reported cstate change: term changed from 3 to 4, leader changed from 333acff728274715a9816c286921ddd0 (127.5.107.68) to 681fd3c49e914094bf28a7df155cffc9 (127.5.107.71). New cstate: current_term: 4 leader_uuid: "681fd3c49e914094bf28a7df155cffc9" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 14:24:18.811187 5927 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:24:18.830724 6057 consensus_peers.cc:485] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:24:18.861565 6057 consensus_peers.cc:485] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:24:18.964231 5797 consensus_peers.cc:485] T 0697cc9c0d26419083f71a411b2699ed P 83e62d4e5f8a4316bc1bcb8c18458081 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:24:19.168776 6122 raft_consensus.cc:1261] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 4 FOLLOWER]: Refusing update from remote peer 681fd3c49e914094bf28a7df155cffc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 4 index: 14. (index mismatch)
I20240502 14:24:19.169512 10406 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 14:24:19.170218 7633 raft_consensus.cc:1261] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e [term 4 FOLLOWER]: Refusing update from remote peer 681fd3c49e914094bf28a7df155cffc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 4 index: 14. (index mismatch)
I20240502 14:24:19.170835 10406 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 14:24:19.177544 7373 raft_consensus.cc:1261] T 5358f204e4324db78adad5af90c17b86 P 806206bb049040bf8db48a7d8d3840c1 [term 4 FOLLOWER]: Refusing update from remote peer 681fd3c49e914094bf28a7df155cffc9: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 4 index: 14. (index mismatch)
I20240502 14:24:19.178207 10381 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 14:24:19.580721 7503 consensus_queue.cc:237] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 4.14, Last appended by leader: 13, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } }
I20240502 14:24:19.582898 7373 raft_consensus.cc:1261] T 5358f204e4324db78adad5af90c17b86 P 806206bb049040bf8db48a7d8d3840c1 [term 4 FOLLOWER]: Refusing update from remote peer 681fd3c49e914094bf28a7df155cffc9: Log matching property violated. Preceding OpId in replica: term: 4 index: 14. Preceding OpId from leader: term: 4 index: 15. (index mismatch)
I20240502 14:24:19.583070 7633 raft_consensus.cc:1261] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e [term 4 FOLLOWER]: Refusing update from remote peer 681fd3c49e914094bf28a7df155cffc9: Log matching property violated. Preceding OpId in replica: term: 4 index: 14. Preceding OpId from leader: term: 4 index: 15. (index mismatch)
I20240502 14:24:19.583439 10406 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240502 14:24:19.583753 10421 consensus_queue.cc:1035] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240502 14:24:19.586196 10421 raft_consensus.cc:2932] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 [term 4 LEADER]: Committing config change with OpId 4.15: config changed from index 13 to 15, VOTER 333acff728274715a9816c286921ddd0 (127.5.107.68) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:24:19.586740 7633 raft_consensus.cc:2932] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e [term 4 FOLLOWER]: Committing config change with OpId 4.15: config changed from index 13 to 15, VOTER 333acff728274715a9816c286921ddd0 (127.5.107.68) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:24:19.590139 7373 raft_consensus.cc:2932] T 5358f204e4324db78adad5af90c17b86 P 806206bb049040bf8db48a7d8d3840c1 [term 4 FOLLOWER]: Committing config change with OpId 4.15: config changed from index 13 to 15, VOTER 333acff728274715a9816c286921ddd0 (127.5.107.68) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:24:19.590768 5579 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5358f204e4324db78adad5af90c17b86 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:24:19.591958 5591 catalog_manager.cc:5484] T 5358f204e4324db78adad5af90c17b86 P 83b0e3c0ba724317832f8505a1ffb66e reported cstate change: config changed from index 13 to 15, VOTER 333acff728274715a9816c286921ddd0 (127.5.107.68) evicted. New cstate: current_term: 4 leader_uuid: "681fd3c49e914094bf28a7df155cffc9" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "83b0e3c0ba724317832f8505a1ffb66e" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35323 } attrs { promote: false } } peers { permanent_uuid: "681fd3c49e914094bf28a7df155cffc9" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 42329 } attrs { promote: false } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: false } } }
I20240502 14:24:19.597463 6103 tablet_service.cc:1514] Processing DeleteTablet for tablet 5358f204e4324db78adad5af90c17b86 with delete_type TABLET_DATA_TOMBSTONED (TS 333acff728274715a9816c286921ddd0 not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:44714
I20240502 14:24:19.598157 10439 tablet_replica.cc:330] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0: stopping tablet replica
I20240502 14:24:19.598380 10439 raft_consensus.cc:2229] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 14:24:19.598574 10439 raft_consensus.cc:2258] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 14:24:19.599432 10439 ts_tablet_manager.cc:1875] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:24:19.605506 10439 ts_tablet_manager.cc:1888] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.14
I20240502 14:24:19.605671 10439 log.cc:1196] T 5358f204e4324db78adad5af90c17b86 P 333acff728274715a9816c286921ddd0: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal/wals/5358f204e4324db78adad5af90c17b86
I20240502 14:24:19.606426 5578 catalog_manager.cc:4830] TS 333acff728274715a9816c286921ddd0 (127.5.107.68:45913): tablet 5358f204e4324db78adad5af90c17b86 (table rebalance_test_table_3 [id=5a9100430c684a96b7fd7563ec356139]) successfully deleted
W20240502 14:24:19.693316 5927 consensus_peers.cc:485] T c3ae6199abe742feabe9832da409cc88 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:24:19.723728 7484 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:24:19.733716 5973 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:24:19.740444 6103 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:24:19.754627 5844 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:24:19.755726 7614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:19.756069 7222 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:24:19.763072 7354 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:24:19.802819 6057 consensus_peers.cc:485] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:24:19.930311 5973 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:24:19.933972 6103 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:24:19.940423 7484 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:24:19.945765 7614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:19.948354 5844 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:24:19.961005 7222 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:24:19.961117 7354 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:24:20.238106 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.65:44831
--local_ip_for_outbound_sockets=127.5.107.65
--tserver_master_addrs=127.5.107.126:34027
--webserver_port=40073
--webserver_interface=127.5.107.65
--builtin_ntp_servers=127.5.107.84:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 14:24:20.548883 10515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:24:20.549142 10515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:24:20.549279 10515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:24:20.555804 10515 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:24:20.555922 10515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:24:20.556063 10515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.65
I20240502 14:24:20.563630 10515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.65:44831
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.107.65
--webserver_port=40073
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.5.107.126:34027
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.65
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:24:20.564759 10515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:24:20.566339 10515 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:24:20.578298 10524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:20.582401 10523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:20.583137 10526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:24:20.583437 10515 server_base.cc:1034] running on GCE node
I20240502 14:24:20.584003 10515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:24:20.584913 10515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:24:20.586104 10515 hybrid_clock.cc:648] HybridClock initialized: now 1714659860586042 us; error 54 us; skew 500 ppm
I20240502 14:24:20.586403 10515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:24:20.588466 10515 webserver.cc:460] Webserver started at http://127.5.107.65:40073/ using document root <none> and password file <none>
I20240502 14:24:20.589148 10515 fs_manager.cc:359] Metadata directory not provided
I20240502 14:24:20.589291 10515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:24:20.594125 10515 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:24:20.597954 10532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:20.599956 10515 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240502 14:24:20.600131 10515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "1dbf482780eb47e8933983e4f0fd0444"
format_stamp: "Formatted at 2024-05-02 14:23:11 on dist-test-slave-2cw1"
I20240502 14:24:20.600939 10515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:24:20.618212 10515 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:24:20.618827 10515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:24:20.619068 10515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:24:20.619825 10515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:24:20.621753 10539 ts_tablet_manager.cc:522] Loading tablet metadata (0/12 complete)
I20240502 14:24:20.643780 10515 ts_tablet_manager.cc:559] Loaded tablet metadata (12 total tablets, 12 live tablets)
I20240502 14:24:20.643853 10515 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.022s user 0.001s sys 0.000s
I20240502 14:24:20.643970 10515 ts_tablet_manager.cc:574] Registering tablets (0/12 complete)
I20240502 14:24:20.646836 10539 tablet_bootstrap.cc:492] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap starting.
I20240502 14:24:20.662348 10515 ts_tablet_manager.cc:590] Registered 12 tablets
I20240502 14:24:20.662446 10515 ts_tablet_manager.cc:569] Time spent register tablets: real 0.018s user 0.017s sys 0.000s
I20240502 14:24:20.664105 10539 log.cc:824] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444: Log is configured to *not* fsync() on all Append() calls
I20240502 14:24:20.697986 10515 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.65:44831
I20240502 14:24:20.698061 10646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.65:44831 every 8 connection(s)
I20240502 14:24:20.699445 10515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 14:24:20.704895 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 10515
I20240502 14:24:20.714691 10647 heartbeater.cc:347] Connected to a master server at 127.5.107.126:34027
I20240502 14:24:20.714951 10647 heartbeater.cc:464] Registering TS with master...
I20240502 14:24:20.715703 10647 heartbeater.cc:511] Master 127.5.107.126:34027 requested a full tablet report, sending...
I20240502 14:24:20.718465 5586 ts_manager.cc:194] Re-registered known tserver with Master: 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831)
I20240502 14:24:20.720335 10539 tablet_bootstrap.cc:492] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:24:20.720993 10539 tablet_bootstrap.cc:492] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap complete.
I20240502 14:24:20.721841 5586 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.65:33967
I20240502 14:24:20.722714 10539 ts_tablet_manager.cc:1367] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444: Time spent bootstrapping tablet: real 0.076s user 0.059s sys 0.013s
I20240502 14:24:20.729251 10539 raft_consensus.cc:348] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:24:20.730185 10539 raft_consensus.cc:729] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbf482780eb47e8933983e4f0fd0444, State: Initialized, Role: FOLLOWER
I20240502 14:24:20.730798 10539 consensus_queue.cc:260] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:24:20.731783 10647 heartbeater.cc:503] Master 127.5.107.126:34027 was elected leader, sending a full tablet report...
I20240502 14:24:20.732314 10539 ts_tablet_manager.cc:1398] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444: Time spent starting tablet: real 0.009s user 0.006s sys 0.004s
I20240502 14:24:20.732641 10539 tablet_bootstrap.cc:492] T 9fdda89a7743466a8aeceb543124fd79 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap starting.
W20240502 14:24:20.759261 5927 consensus_peers.cc:485] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 101: this message will repeat every 5th retry.
I20240502 14:24:20.780470 10539 tablet_bootstrap.cc:492] T 9fdda89a7743466a8aeceb543124fd79 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:24:20.780942 10539 tablet_bootstrap.cc:492] T 9fdda89a7743466a8aeceb543124fd79 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap complete.
I20240502 14:24:20.781980 10539 ts_tablet_manager.cc:1367] T 9fdda89a7743466a8aeceb543124fd79 P 1dbf482780eb47e8933983e4f0fd0444: Time spent bootstrapping tablet: real 0.049s user 0.036s sys 0.011s
I20240502 14:24:20.782706 10539 raft_consensus.cc:348] T 9fdda89a7743466a8aeceb543124fd79 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:24:20.782862 10539 raft_consensus.cc:729] T 9fdda89a7743466a8aeceb543124fd79 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbf482780eb47e8933983e4f0fd0444, State: Initialized, Role: FOLLOWER
I20240502 14:24:20.783064 10539 consensus_queue.cc:260] T 9fdda89a7743466a8aeceb543124fd79 P 1dbf482780eb47e8933983e4f0fd0444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:24:20.783507 10539 ts_tablet_manager.cc:1398] T 9fdda89a7743466a8aeceb543124fd79 P 1dbf482780eb47e8933983e4f0fd0444: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:24:20.783776 10539 tablet_bootstrap.cc:492] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap starting.
I20240502 14:24:20.839519 10539 tablet_bootstrap.cc:492] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:24:20.840199 10539 tablet_bootstrap.cc:492] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap complete.
I20240502 14:24:20.841771 10539 ts_tablet_manager.cc:1367] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444: Time spent bootstrapping tablet: real 0.058s user 0.040s sys 0.015s
I20240502 14:24:20.842579 10539 raft_consensus.cc:348] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } }
I20240502 14:24:20.842797 10539 raft_consensus.cc:729] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbf482780eb47e8933983e4f0fd0444, State: Initialized, Role: FOLLOWER
I20240502 14:24:20.843098 10539 consensus_queue.cc:260] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "806206bb049040bf8db48a7d8d3840c1" member_type: NON_VOTER last_known_addr { host: "127.5.107.70" port: 40027 } attrs { promote: true } }
I20240502 14:24:20.843663 10539 ts_tablet_manager.cc:1398] T a6647f23da6c4430a9f84e378dd345a4 P 1dbf482780eb47e8933983e4f0fd0444: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:24:20.843911 10539 tablet_bootstrap.cc:492] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap starting.
I20240502 14:24:20.846725 10582 tablet_service.cc:1514] Processing DeleteTablet for tablet aca9200e5c264c27833d7a4a9122c4d9 with delete_type TABLET_DATA_TOMBSTONED (TS 1dbf482780eb47e8933983e4f0fd0444 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:51358
W20240502 14:24:20.849813 5576 catalog_manager.cc:4810] TS 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): delete failed for tablet aca9200e5c264c27833d7a4a9122c4d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aca9200e5c264c27833d7a4a9122c4d9 already in progress: opening tablet
I20240502 14:24:20.879451 10539 tablet_bootstrap.cc:492] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:24:20.879869 10539 tablet_bootstrap.cc:492] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap complete.
I20240502 14:24:20.880823 10539 ts_tablet_manager.cc:1367] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444: Time spent bootstrapping tablet: real 0.037s user 0.033s sys 0.000s
I20240502 14:24:20.881508 10539 raft_consensus.cc:348] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:24:20.881687 10539 raft_consensus.cc:729] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbf482780eb47e8933983e4f0fd0444, State: Initialized, Role: FOLLOWER
I20240502 14:24:20.881968 10539 consensus_queue.cc:260] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:24:20.882411 10539 ts_tablet_manager.cc:1398] T 25f9ebd715dc4f96901b60796ea846ad P 1dbf482780eb47e8933983e4f0fd0444: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 14:24:20.882633 10539 tablet_bootstrap.cc:492] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap starting.
I20240502 14:24:20.920235 10539 tablet_bootstrap.cc:492] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:24:20.920648 10539 tablet_bootstrap.cc:492] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap complete.
I20240502 14:24:20.921772 10539 ts_tablet_manager.cc:1367] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444: Time spent bootstrapping tablet: real 0.039s user 0.027s sys 0.011s
I20240502 14:24:20.922433 10539 raft_consensus.cc:348] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:24:20.922605 10539 raft_consensus.cc:729] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbf482780eb47e8933983e4f0fd0444, State: Initialized, Role: FOLLOWER
I20240502 14:24:20.922860 10539 consensus_queue.cc:260] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:24:20.923393 10539 ts_tablet_manager.cc:1398] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:24:20.923636 10539 tablet_bootstrap.cc:492] T 7d944a63fdf24ecd8fadbad3b620af0a P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap starting.
I20240502 14:24:20.941870 10602 raft_consensus.cc:3037] T 0697cc9c0d26419083f71a411b2699ed P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:24:20.975467 10539 tablet_bootstrap.cc:492] T 7d944a63fdf24ecd8fadbad3b620af0a P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:24:20.975855 10539 tablet_bootstrap.cc:492] T 7d944a63fdf24ecd8fadbad3b620af0a P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap complete.
I20240502 14:24:20.976688 10539 ts_tablet_manager.cc:1367] T 7d944a63fdf24ecd8fadbad3b620af0a P 1dbf482780eb47e8933983e4f0fd0444: Time spent bootstrapping tablet: real 0.053s user 0.036s sys 0.012s
I20240502 14:24:20.977339 10539 raft_consensus.cc:348] T 7d944a63fdf24ecd8fadbad3b620af0a P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:24:20.977531 10539 raft_consensus.cc:729] T 7d944a63fdf24ecd8fadbad3b620af0a P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbf482780eb47e8933983e4f0fd0444, State: Initialized, Role: FOLLOWER
I20240502 14:24:20.977802 10539 consensus_queue.cc:260] T 7d944a63fdf24ecd8fadbad3b620af0a P 1dbf482780eb47e8933983e4f0fd0444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } }
I20240502 14:24:20.978315 10539 ts_tablet_manager.cc:1398] T 7d944a63fdf24ecd8fadbad3b620af0a P 1dbf482780eb47e8933983e4f0fd0444: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240502 14:24:20.978564 10539 tablet_bootstrap.cc:492] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap starting.
I20240502 14:24:21.014411 10539 tablet_bootstrap.cc:492] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:24:21.014819 10539 tablet_bootstrap.cc:492] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap complete.
I20240502 14:24:21.015678 10539 ts_tablet_manager.cc:1367] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444: Time spent bootstrapping tablet: real 0.037s user 0.027s sys 0.008s
I20240502 14:24:21.016249 10539 raft_consensus.cc:348] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:24:21.016453 10539 raft_consensus.cc:729] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbf482780eb47e8933983e4f0fd0444, State: Initialized, Role: FOLLOWER
I20240502 14:24:21.016678 10539 consensus_queue.cc:260] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:24:21.017141 10539 ts_tablet_manager.cc:1398] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:24:21.017439 10539 tablet_bootstrap.cc:492] T 228133f033a44f59af1259cc1a1e9d55 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap starting.
I20240502 14:24:21.019878 10582 tablet_service.cc:1514] Processing DeleteTablet for tablet aca9200e5c264c27833d7a4a9122c4d9 with delete_type TABLET_DATA_TOMBSTONED (Replica has no consensus available (current committed config index is 11)) from {username='slave'} at 127.0.0.1:51358
I20240502 14:24:21.020295 10659 tablet_replica.cc:330] stopping tablet replica
I20240502 14:24:21.020779 10659 raft_consensus.cc:2229] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:24:21.021149 10659 raft_consensus.cc:2258] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:24:21.022536 10659 ts_tablet_manager.cc:1875] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:24:21.030138 10659 ts_tablet_manager.cc:1888] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:24:21.030303 10659 log.cc:1196] T aca9200e5c264c27833d7a4a9122c4d9 P 1dbf482780eb47e8933983e4f0fd0444: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal/wals/aca9200e5c264c27833d7a4a9122c4d9
I20240502 14:24:21.031194 5576 catalog_manager.cc:4830] TS 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): tablet aca9200e5c264c27833d7a4a9122c4d9 (table rebalance_test_table_0 [id=7a7408907bfc4fd2bd39cf4e8b4c97fb]) successfully deleted
I20240502 14:24:21.051102 10539 tablet_bootstrap.cc:492] T 228133f033a44f59af1259cc1a1e9d55 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:24:21.051635 10539 tablet_bootstrap.cc:492] T 228133f033a44f59af1259cc1a1e9d55 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap complete.
I20240502 14:24:21.052603 10539 ts_tablet_manager.cc:1367] T 228133f033a44f59af1259cc1a1e9d55 P 1dbf482780eb47e8933983e4f0fd0444: Time spent bootstrapping tablet: real 0.035s user 0.029s sys 0.004s
I20240502 14:24:21.053172 10539 raft_consensus.cc:348] T 228133f033a44f59af1259cc1a1e9d55 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:24:21.053357 10539 raft_consensus.cc:729] T 228133f033a44f59af1259cc1a1e9d55 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbf482780eb47e8933983e4f0fd0444, State: Initialized, Role: FOLLOWER
I20240502 14:24:21.053609 10539 consensus_queue.cc:260] T 228133f033a44f59af1259cc1a1e9d55 P 1dbf482780eb47e8933983e4f0fd0444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } }
I20240502 14:24:21.054033 10539 ts_tablet_manager.cc:1398] T 228133f033a44f59af1259cc1a1e9d55 P 1dbf482780eb47e8933983e4f0fd0444: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:24:21.054294 10539 tablet_bootstrap.cc:492] T 2f624fefb7de4083b61b38f7b09d3881 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap starting.
I20240502 14:24:21.092415 10539 tablet_bootstrap.cc:492] T 2f624fefb7de4083b61b38f7b09d3881 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:24:21.092871 10539 tablet_bootstrap.cc:492] T 2f624fefb7de4083b61b38f7b09d3881 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap complete.
I20240502 14:24:21.093767 10539 ts_tablet_manager.cc:1367] T 2f624fefb7de4083b61b38f7b09d3881 P 1dbf482780eb47e8933983e4f0fd0444: Time spent bootstrapping tablet: real 0.040s user 0.031s sys 0.007s
I20240502 14:24:21.094370 10539 raft_consensus.cc:348] T 2f624fefb7de4083b61b38f7b09d3881 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } }
I20240502 14:24:21.094554 10539 raft_consensus.cc:729] T 2f624fefb7de4083b61b38f7b09d3881 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbf482780eb47e8933983e4f0fd0444, State: Initialized, Role: FOLLOWER
I20240502 14:24:21.094929 10539 consensus_queue.cc:260] T 2f624fefb7de4083b61b38f7b09d3881 P 1dbf482780eb47e8933983e4f0fd0444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } }
I20240502 14:24:21.095431 10539 ts_tablet_manager.cc:1398] T 2f624fefb7de4083b61b38f7b09d3881 P 1dbf482780eb47e8933983e4f0fd0444: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:24:21.095680 10539 tablet_bootstrap.cc:492] T f6a8b93193d64330bc53a5c3b363f695 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap starting.
I20240502 14:24:21.129264 10539 tablet_bootstrap.cc:492] T f6a8b93193d64330bc53a5c3b363f695 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:24:21.129812 10539 tablet_bootstrap.cc:492] T f6a8b93193d64330bc53a5c3b363f695 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap complete.
I20240502 14:24:21.130884 10539 ts_tablet_manager.cc:1367] T f6a8b93193d64330bc53a5c3b363f695 P 1dbf482780eb47e8933983e4f0fd0444: Time spent bootstrapping tablet: real 0.035s user 0.022s sys 0.012s
I20240502 14:24:21.131712 10539 raft_consensus.cc:348] T f6a8b93193d64330bc53a5c3b363f695 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } }
I20240502 14:24:21.131942 10539 raft_consensus.cc:729] T f6a8b93193d64330bc53a5c3b363f695 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbf482780eb47e8933983e4f0fd0444, State: Initialized, Role: FOLLOWER
I20240502 14:24:21.132192 10539 consensus_queue.cc:260] T f6a8b93193d64330bc53a5c3b363f695 P 1dbf482780eb47e8933983e4f0fd0444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "333acff728274715a9816c286921ddd0" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 45913 } }
I20240502 14:24:21.132787 10539 ts_tablet_manager.cc:1398] T f6a8b93193d64330bc53a5c3b363f695 P 1dbf482780eb47e8933983e4f0fd0444: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20240502 14:24:21.133057 10539 tablet_bootstrap.cc:492] T c3ae6199abe742feabe9832da409cc88 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap starting.
I20240502 14:24:21.181567 10539 tablet_bootstrap.cc:492] T c3ae6199abe742feabe9832da409cc88 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:24:21.182274 10539 tablet_bootstrap.cc:492] T c3ae6199abe742feabe9832da409cc88 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap complete.
I20240502 14:24:21.183431 10539 ts_tablet_manager.cc:1367] T c3ae6199abe742feabe9832da409cc88 P 1dbf482780eb47e8933983e4f0fd0444: Time spent bootstrapping tablet: real 0.050s user 0.046s sys 0.003s
I20240502 14:24:21.184190 10539 raft_consensus.cc:348] T c3ae6199abe742feabe9832da409cc88 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:24:21.184500 10539 raft_consensus.cc:729] T c3ae6199abe742feabe9832da409cc88 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbf482780eb47e8933983e4f0fd0444, State: Initialized, Role: FOLLOWER
I20240502 14:24:21.184828 10539 consensus_queue.cc:260] T c3ae6199abe742feabe9832da409cc88 P 1dbf482780eb47e8933983e4f0fd0444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:24:21.186360 10539 ts_tablet_manager.cc:1398] T c3ae6199abe742feabe9832da409cc88 P 1dbf482780eb47e8933983e4f0fd0444: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:24:21.186574 10539 tablet_bootstrap.cc:492] T 3128a2a3296e4940bfdf669fecfd7877 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap starting.
I20240502 14:24:21.236528 10582 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:21.257102 10539 tablet_bootstrap.cc:492] T 3128a2a3296e4940bfdf669fecfd7877 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:24:21.257826 10539 tablet_bootstrap.cc:492] T 3128a2a3296e4940bfdf669fecfd7877 P 1dbf482780eb47e8933983e4f0fd0444: Bootstrap complete.
I20240502 14:24:21.259172 10539 ts_tablet_manager.cc:1367] T 3128a2a3296e4940bfdf669fecfd7877 P 1dbf482780eb47e8933983e4f0fd0444: Time spent bootstrapping tablet: real 0.073s user 0.062s sys 0.008s
I20240502 14:24:21.260162 10539 raft_consensus.cc:348] T 3128a2a3296e4940bfdf669fecfd7877 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:24:21.260424 10539 raft_consensus.cc:729] T 3128a2a3296e4940bfdf669fecfd7877 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbf482780eb47e8933983e4f0fd0444, State: Initialized, Role: FOLLOWER
I20240502 14:24:21.260769 10539 consensus_queue.cc:260] T 3128a2a3296e4940bfdf669fecfd7877 P 1dbf482780eb47e8933983e4f0fd0444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e62d4e5f8a4316bc1bcb8c18458081" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 34015 } } peers { permanent_uuid: "181b466de0084fcbace0ca327fb1cbef" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 41517 } } peers { permanent_uuid: "1dbf482780eb47e8933983e4f0fd0444" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 44831 } }
I20240502 14:24:21.261758 10539 ts_tablet_manager.cc:1398] T 3128a2a3296e4940bfdf669fecfd7877 P 1dbf482780eb47e8933983e4f0fd0444: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:24:21.266574 7484 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:24:21.273388 5973 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:24:21.308259 10602 raft_consensus.cc:3037] T 244218906a51489dab769122731b1ec2 P 1dbf482780eb47e8933983e4f0fd0444 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:24:21.310999 6103 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:24:21.315485 7222 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:24:21.335230 7354 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:24:21.357074 5844 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:24:21.369056 7614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:21.557684 10582 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:21.571923 7484 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:24:21.587312 5973 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:24:21.587600 6103 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:24:21.595388 7222 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:24:21.632972 7354 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:24:21.636170 7614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:21.662015 5844 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:24:21.802870 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 10515
I20240502 14:24:21.812206 10641 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:24:21.922299 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 10515
W20240502 14:24:21.945931 5927 connection.cc:535] client connection to 127.5.107.65:44831 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:24:21.946099 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 5782
W20240502 14:24:21.947935 5927 consensus_peers.cc:485] T 3128a2a3296e4940bfdf669fecfd7877 P 181b466de0084fcbace0ca327fb1cbef -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:24:21.951262 5903 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:24:22.123696 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 5782
W20240502 14:24:22.152197 7568 connection.cc:535] server connection from 127.5.107.66:38481 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:24:22.153262 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 5912
I20240502 14:24:22.158185 6033 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 14:24:22.216279 6057 consensus_peers.cc:485] T f6a8b93193d64330bc53a5c3b363f695 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:24:22.279572 6057 consensus_peers.cc:485] T 244218906a51489dab769122731b1ec2 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:24:22.311481 6057 consensus_peers.cc:485] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 -> Peer 1dbf482780eb47e8933983e4f0fd0444 (127.5.107.65:44831): Couldn't send request to peer 1dbf482780eb47e8933983e4f0fd0444. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.65:44831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:24:22.333240 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 5912
W20240502 14:24:22.361116 7438 connection.cc:535] server connection from 127.5.107.67:37457 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:24:22.361488 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 6042
I20240502 14:24:22.367583 6163 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 14:24:22.570596 6059 consensus_peers.cc:485] T 2f624fefb7de4083b61b38f7b09d3881 P 333acff728274715a9816c286921ddd0 -> Peer 181b466de0084fcbace0ca327fb1cbef (127.5.107.67:41517): Couldn't send request to peer 181b466de0084fcbace0ca327fb1cbef. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.67:41517: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:24:22.571002 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 6042
W20240502 14:24:22.573560 7309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.67:41517: connect: Connection refused (error 111)
W20240502 14:24:22.578228 7309 consensus_peers.cc:485] T 9fdda89a7743466a8aeceb543124fd79 P 806206bb049040bf8db48a7d8d3840c1 -> Peer 181b466de0084fcbace0ca327fb1cbef (127.5.107.67:41517): Couldn't send request to peer 181b466de0084fcbace0ca327fb1cbef. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.67:41517: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:24:22.606746 7439 connection.cc:535] server connection from 127.5.107.68:60595 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:24:22.607388 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 7160
I20240502 14:24:22.613137 7282 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:24:22.734555 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 7160
I20240502 14:24:22.756686 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 7291
W20240502 14:24:22.756650 5576 connection.cc:535] server connection from 127.5.107.69:58265 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:24:22.763043 7413 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:24:22.895290 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 7291
W20240502 14:24:22.918124 7439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.107.70:40027 (error 108)
I20240502 14:24:22.918586 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 7422
W20240502 14:24:22.920526 7439 consensus_peers.cc:485] T 5358f204e4324db78adad5af90c17b86 P 681fd3c49e914094bf28a7df155cffc9 -> Peer 806206bb049040bf8db48a7d8d3840c1 (127.5.107.70:40027): Couldn't send request to peer 806206bb049040bf8db48a7d8d3840c1. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:40027: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:24:22.924883 7542 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:24:23.049394 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 7422
I20240502 14:24:23.071764 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 7552
I20240502 14:24:23.077208 7672 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:24:23.202198 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 7552
I20240502 14:24:23.225431 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 5561
I20240502 14:24:23.230823 5621 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:24:23.400082 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 5561
2024-05-02T14:24:23Z chronyd exiting
[ OK ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown (72969 ms)
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest (72969 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0
I20240502 14:24:23.471393 5549 test_util.cc:268] Using random seed: 1544725760
I20240502 14:24:23.473220 5549 ts_itest-base.cc:112] Starting cluster with:
I20240502 14:24:23.473290 5549 ts_itest-base.cc:113] --------------
I20240502 14:24:23.473340 5549 ts_itest-base.cc:114] 5 tablet servers
I20240502 14:24:23.473395 5549 ts_itest-base.cc:115] 3 replicas per TS
I20240502 14:24:23.473439 5549 ts_itest-base.cc:116] --------------
2024-05-02T14:24:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:24:23Z Disabled control of system clock
I20240502 14:24:23.495136 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.107.126:40403
--webserver_interface=127.5.107.126
--webserver_port=0
--builtin_ntp_servers=127.5.107.84:36267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.107.126:40403
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240502 14:24:23.789499 10798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:24:23.789774 10798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:24:23.789901 10798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:24:23.796268 10798 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:24:23.796381 10798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:24:23.796448 10798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:24:23.796522 10798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:24:23.796579 10798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:24:23.804267 10798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:36267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.5.107.126:40403
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.107.126:40403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.107.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:24:23.805356 10798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:24:23.806901 10798 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:24:23.817045 10806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:23.817226 10803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:23.817695 10804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:24:23.818272 10798 server_base.cc:1034] running on GCE node
I20240502 14:24:23.818918 10798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:24:23.820113 10798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:24:23.821269 10798 hybrid_clock.cc:648] HybridClock initialized: now 1714659863821225 us; error 28 us; skew 500 ppm
I20240502 14:24:23.821558 10798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:24:23.827874 10798 webserver.cc:460] Webserver started at http://127.5.107.126:40901/ using document root <none> and password file <none>
I20240502 14:24:23.828510 10798 fs_manager.cc:359] Metadata directory not provided
I20240502 14:24:23.828603 10798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:24:23.828892 10798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:24:23.831372 10798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "83beb625485d4443883dcc1d12d41272"
format_stamp: "Formatted at 2024-05-02 14:24:23 on dist-test-slave-2cw1"
I20240502 14:24:23.832080 10798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "83beb625485d4443883dcc1d12d41272"
format_stamp: "Formatted at 2024-05-02 14:24:23 on dist-test-slave-2cw1"
I20240502 14:24:23.836928 10798 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20240502 14:24:23.839955 10812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:23.841164 10798 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:24:23.841378 10798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/wal
uuid: "83beb625485d4443883dcc1d12d41272"
format_stamp: "Formatted at 2024-05-02 14:24:23 on dist-test-slave-2cw1"
I20240502 14:24:23.841555 10798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:24:23.858285 10798 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:24:23.858968 10798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:24:23.859187 10798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:24:23.875540 10798 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.126:40403
I20240502 14:24:23.875555 10863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.126:40403 every 8 connection(s)
I20240502 14:24:23.876737 10798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 14:24:23.877753 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 10798
I20240502 14:24:23.877966 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 14:24:23.880857 10864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:23.890838 10864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83beb625485d4443883dcc1d12d41272: Bootstrap starting.
I20240502 14:24:23.893479 10864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83beb625485d4443883dcc1d12d41272: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:23.894289 10864 log.cc:824] T 00000000000000000000000000000000 P 83beb625485d4443883dcc1d12d41272: Log is configured to *not* fsync() on all Append() calls
I20240502 14:24:23.896231 10864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83beb625485d4443883dcc1d12d41272: No bootstrap required, opened a new log
I20240502 14:24:23.901225 10864 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83beb625485d4443883dcc1d12d41272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83beb625485d4443883dcc1d12d41272" member_type: VOTER last_known_addr { host: "127.5.107.126" port: 40403 } }
I20240502 14:24:23.901541 10864 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83beb625485d4443883dcc1d12d41272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:23.901629 10864 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83beb625485d4443883dcc1d12d41272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83beb625485d4443883dcc1d12d41272, State: Initialized, Role: FOLLOWER
I20240502 14:24:23.902140 10864 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83beb625485d4443883dcc1d12d41272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83beb625485d4443883dcc1d12d41272" member_type: VOTER last_known_addr { host: "127.5.107.126" port: 40403 } }
I20240502 14:24:23.902324 10864 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83beb625485d4443883dcc1d12d41272 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:24:23.902451 10864 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83beb625485d4443883dcc1d12d41272 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:24:23.902581 10864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83beb625485d4443883dcc1d12d41272 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:23.904372 10864 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83beb625485d4443883dcc1d12d41272 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83beb625485d4443883dcc1d12d41272" member_type: VOTER last_known_addr { host: "127.5.107.126" port: 40403 } }
I20240502 14:24:23.904742 10864 leader_election.cc:302] T 00000000000000000000000000000000 P 83beb625485d4443883dcc1d12d41272 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83beb625485d4443883dcc1d12d41272; no voters:
I20240502 14:24:23.905225 10864 leader_election.cc:288] T 00000000000000000000000000000000 P 83beb625485d4443883dcc1d12d41272 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:24:23.905373 10869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83beb625485d4443883dcc1d12d41272 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:24:23.905910 10869 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83beb625485d4443883dcc1d12d41272 [term 1 LEADER]: Becoming Leader. State: Replica: 83beb625485d4443883dcc1d12d41272, State: Running, Role: LEADER
I20240502 14:24:23.906396 10869 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83beb625485d4443883dcc1d12d41272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83beb625485d4443883dcc1d12d41272" member_type: VOTER last_known_addr { host: "127.5.107.126" port: 40403 } }
I20240502 14:24:23.906890 10864 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83beb625485d4443883dcc1d12d41272 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:24:23.908857 10871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83beb625485d4443883dcc1d12d41272 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83beb625485d4443883dcc1d12d41272. Latest consensus state: current_term: 1 leader_uuid: "83beb625485d4443883dcc1d12d41272" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83beb625485d4443883dcc1d12d41272" member_type: VOTER last_known_addr { host: "127.5.107.126" port: 40403 } } }
I20240502 14:24:23.908800 10870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83beb625485d4443883dcc1d12d41272 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83beb625485d4443883dcc1d12d41272" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83beb625485d4443883dcc1d12d41272" member_type: VOTER last_known_addr { host: "127.5.107.126" port: 40403 } } }
I20240502 14:24:23.909197 10871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83beb625485d4443883dcc1d12d41272 [sys.catalog]: This master's current role is: LEADER
I20240502 14:24:23.909225 10870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83beb625485d4443883dcc1d12d41272 [sys.catalog]: This master's current role is: LEADER
I20240502 14:24:23.910873 10876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:24:23.915601 10876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:24:23.923276 10876 catalog_manager.cc:1347] Generated new cluster ID: 68545309d702407f95d2a5dbd04a4992
I20240502 14:24:23.923378 10876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:24:23.938935 10876 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:24:23.940021 10876 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:24:23.951493 10876 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 83beb625485d4443883dcc1d12d41272: Generated new TSK 0
I20240502 14:24:23.952232 10876 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:24:23.964571 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.65:0
--local_ip_for_outbound_sockets=127.5.107.65
--webserver_interface=127.5.107.65
--webserver_port=0
--tserver_master_addrs=127.5.107.126:40403
--builtin_ntp_servers=127.5.107.84:36267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 14:24:24.267112 10888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:24:24.267372 10888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:24:24.267503 10888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:24:24.274578 10888 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:24:24.274698 10888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:24:24.274835 10888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.65
I20240502 14:24:24.282869 10888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:36267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.107.65
--webserver_port=0
--tserver_master_addrs=127.5.107.126:40403
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.65
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:24:24.283970 10888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:24:24.285475 10888 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:24:24.296123 10893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:24.296717 10894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:24.297346 10896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:24:24.297453 10888 server_base.cc:1034] running on GCE node
I20240502 14:24:24.297988 10888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:24:24.298837 10888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:24:24.300040 10888 hybrid_clock.cc:648] HybridClock initialized: now 1714659864299994 us; error 34 us; skew 500 ppm
I20240502 14:24:24.300312 10888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:24:24.302502 10888 webserver.cc:460] Webserver started at http://127.5.107.65:33335/ using document root <none> and password file <none>
I20240502 14:24:24.303170 10888 fs_manager.cc:359] Metadata directory not provided
I20240502 14:24:24.303273 10888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:24:24.303582 10888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:24:24.308039 10888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "60fa049c8dc94ca7935b892be97a7f80"
format_stamp: "Formatted at 2024-05-02 14:24:24 on dist-test-slave-2cw1"
I20240502 14:24:24.308701 10888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "60fa049c8dc94ca7935b892be97a7f80"
format_stamp: "Formatted at 2024-05-02 14:24:24 on dist-test-slave-2cw1"
I20240502 14:24:24.313366 10888 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20240502 14:24:24.316170 10902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:24.317251 10888 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:24:24.317387 10888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "60fa049c8dc94ca7935b892be97a7f80"
format_stamp: "Formatted at 2024-05-02 14:24:24 on dist-test-slave-2cw1"
I20240502 14:24:24.317575 10888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:24:24.333410 10888 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:24:24.334115 10888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:24:24.334347 10888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:24:24.335139 10888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:24:24.336478 10888 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:24:24.336572 10888 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:24.336673 10888 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:24:24.336735 10888 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:24.364071 10888 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.65:40283
I20240502 14:24:24.364114 11014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.65:40283 every 8 connection(s)
I20240502 14:24:24.365238 10888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 14:24:24.373780 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 10888
I20240502 14:24:24.373989 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 14:24:24.376747 11015 heartbeater.cc:347] Connected to a master server at 127.5.107.126:40403
I20240502 14:24:24.376986 11015 heartbeater.cc:464] Registering TS with master...
I20240502 14:24:24.377167 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.66:0
--local_ip_for_outbound_sockets=127.5.107.66
--webserver_interface=127.5.107.66
--webserver_port=0
--tserver_master_addrs=127.5.107.126:40403
--builtin_ntp_servers=127.5.107.84:36267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:24:24.377657 11015 heartbeater.cc:511] Master 127.5.107.126:40403 requested a full tablet report, sending...
I20240502 14:24:24.379316 10828 ts_manager.cc:194] Registered new tserver with Master: 60fa049c8dc94ca7935b892be97a7f80 (127.5.107.65:40283)
I20240502 14:24:24.380712 10828 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.65:44345
W20240502 14:24:24.665895 11018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:24:24.666113 11018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:24:24.666224 11018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:24:24.672832 11018 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:24:24.672923 11018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:24:24.673070 11018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.66
I20240502 14:24:24.680308 11018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:36267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.107.66
--webserver_port=0
--tserver_master_addrs=127.5.107.126:40403
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.66
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:24:24.681385 11018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:24:24.682943 11018 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:24:24.692548 11023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:24.693197 11024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:24.693701 11026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:24:24.693982 11018 server_base.cc:1034] running on GCE node
I20240502 14:24:24.694447 11018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:24:24.695175 11018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:24:24.696346 11018 hybrid_clock.cc:648] HybridClock initialized: now 1714659864696304 us; error 35 us; skew 500 ppm
I20240502 14:24:24.696595 11018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:24:24.698448 11018 webserver.cc:460] Webserver started at http://127.5.107.66:46017/ using document root <none> and password file <none>
I20240502 14:24:24.699079 11018 fs_manager.cc:359] Metadata directory not provided
I20240502 14:24:24.699174 11018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:24:24.699432 11018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:24:24.701691 11018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "af87759abc1044aa871c7201a9a1559a"
format_stamp: "Formatted at 2024-05-02 14:24:24 on dist-test-slave-2cw1"
I20240502 14:24:24.702404 11018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "af87759abc1044aa871c7201a9a1559a"
format_stamp: "Formatted at 2024-05-02 14:24:24 on dist-test-slave-2cw1"
I20240502 14:24:24.706641 11018 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 14:24:24.709434 11032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:24.710451 11018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:24:24.710577 11018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "af87759abc1044aa871c7201a9a1559a"
format_stamp: "Formatted at 2024-05-02 14:24:24 on dist-test-slave-2cw1"
I20240502 14:24:24.710745 11018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:24:24.733091 11018 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:24:24.733779 11018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:24:24.734025 11018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:24:24.734858 11018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:24:24.736204 11018 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:24:24.736302 11018 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:24.736421 11018 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:24:24.736495 11018 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:24.766853 11018 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.66:39413
I20240502 14:24:24.766907 11144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.66:39413 every 8 connection(s)
I20240502 14:24:24.767975 11018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 14:24:24.772397 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 11018
I20240502 14:24:24.772609 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 14:24:24.775202 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.67:0
--local_ip_for_outbound_sockets=127.5.107.67
--webserver_interface=127.5.107.67
--webserver_port=0
--tserver_master_addrs=127.5.107.126:40403
--builtin_ntp_servers=127.5.107.84:36267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:24:24.778971 11145 heartbeater.cc:347] Connected to a master server at 127.5.107.126:40403
I20240502 14:24:24.779217 11145 heartbeater.cc:464] Registering TS with master...
I20240502 14:24:24.780001 11145 heartbeater.cc:511] Master 127.5.107.126:40403 requested a full tablet report, sending...
I20240502 14:24:24.781281 10828 ts_manager.cc:194] Registered new tserver with Master: af87759abc1044aa871c7201a9a1559a (127.5.107.66:39413)
I20240502 14:24:24.782023 10828 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.66:54549
W20240502 14:24:25.057686 11148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:24:25.057924 11148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:24:25.058048 11148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:24:25.064749 11148 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:24:25.064846 11148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:24:25.064991 11148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.67
I20240502 14:24:25.072831 11148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:36267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.107.67
--webserver_port=0
--tserver_master_addrs=127.5.107.126:40403
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.67
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:24:25.073894 11148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:24:25.075493 11148 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:24:25.085887 11153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:25.086442 11154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:25.087065 11156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:24:25.087307 11148 server_base.cc:1034] running on GCE node
I20240502 14:24:25.087750 11148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:24:25.088461 11148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:24:25.089638 11148 hybrid_clock.cc:648] HybridClock initialized: now 1714659865089605 us; error 32 us; skew 500 ppm
I20240502 14:24:25.089885 11148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:24:25.091657 11148 webserver.cc:460] Webserver started at http://127.5.107.67:39041/ using document root <none> and password file <none>
I20240502 14:24:25.092289 11148 fs_manager.cc:359] Metadata directory not provided
I20240502 14:24:25.092394 11148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:24:25.092702 11148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:24:25.094782 11148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "766502fe87a248d78a7150f2b40c11de"
format_stamp: "Formatted at 2024-05-02 14:24:25 on dist-test-slave-2cw1"
I20240502 14:24:25.095412 11148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "766502fe87a248d78a7150f2b40c11de"
format_stamp: "Formatted at 2024-05-02 14:24:25 on dist-test-slave-2cw1"
I20240502 14:24:25.099550 11148 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20240502 14:24:25.102622 11162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:25.103615 11148 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:24:25.103751 11148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "766502fe87a248d78a7150f2b40c11de"
format_stamp: "Formatted at 2024-05-02 14:24:25 on dist-test-slave-2cw1"
I20240502 14:24:25.103904 11148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:24:25.124708 11148 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:24:25.125382 11148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:24:25.125602 11148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:24:25.126416 11148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:24:25.127609 11148 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:24:25.127696 11148 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:25.127784 11148 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:24:25.127843 11148 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:25.154794 11148 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.67:32873
I20240502 14:24:25.154841 11275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.67:32873 every 8 connection(s)
I20240502 14:24:25.155861 11148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 14:24:25.160765 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 11148
I20240502 14:24:25.160943 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 14:24:25.164139 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.68:0
--local_ip_for_outbound_sockets=127.5.107.68
--webserver_interface=127.5.107.68
--webserver_port=0
--tserver_master_addrs=127.5.107.126:40403
--builtin_ntp_servers=127.5.107.84:36267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:24:25.167269 11276 heartbeater.cc:347] Connected to a master server at 127.5.107.126:40403
I20240502 14:24:25.167527 11276 heartbeater.cc:464] Registering TS with master...
I20240502 14:24:25.168247 11276 heartbeater.cc:511] Master 127.5.107.126:40403 requested a full tablet report, sending...
I20240502 14:24:25.169615 10828 ts_manager.cc:194] Registered new tserver with Master: 766502fe87a248d78a7150f2b40c11de (127.5.107.67:32873)
I20240502 14:24:25.170686 10828 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.67:44197
I20240502 14:24:25.383208 11015 heartbeater.cc:503] Master 127.5.107.126:40403 was elected leader, sending a full tablet report...
W20240502 14:24:25.453531 11279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:24:25.453753 11279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:24:25.453863 11279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:24:25.459998 11279 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:24:25.460085 11279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:24:25.460196 11279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.68
I20240502 14:24:25.467628 11279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:36267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.5.107.68
--webserver_port=0
--tserver_master_addrs=127.5.107.126:40403
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.68
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:24:25.468648 11279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:24:25.470137 11279 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:24:25.480134 11284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:25.480713 11287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:25.481350 11285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:24:25.481964 11279 server_base.cc:1034] running on GCE node
I20240502 14:24:25.482447 11279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:24:25.483213 11279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:24:25.484396 11279 hybrid_clock.cc:648] HybridClock initialized: now 1714659865484361 us; error 27 us; skew 500 ppm
I20240502 14:24:25.484649 11279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:24:25.486413 11279 webserver.cc:460] Webserver started at http://127.5.107.68:39793/ using document root <none> and password file <none>
I20240502 14:24:25.486968 11279 fs_manager.cc:359] Metadata directory not provided
I20240502 14:24:25.487058 11279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:24:25.487310 11279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:24:25.489375 11279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "b14f5c5b988b4542a2e1b71620e913de"
format_stamp: "Formatted at 2024-05-02 14:24:25 on dist-test-slave-2cw1"
I20240502 14:24:25.489996 11279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "b14f5c5b988b4542a2e1b71620e913de"
format_stamp: "Formatted at 2024-05-02 14:24:25 on dist-test-slave-2cw1"
I20240502 14:24:25.494215 11279 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 14:24:25.496944 11293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:25.498037 11279 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:24:25.498178 11279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "b14f5c5b988b4542a2e1b71620e913de"
format_stamp: "Formatted at 2024-05-02 14:24:25 on dist-test-slave-2cw1"
I20240502 14:24:25.498332 11279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:24:25.536842 11279 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:24:25.537662 11279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:24:25.537930 11279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:24:25.538815 11279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:24:25.540174 11279 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:24:25.540268 11279 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:25.540385 11279 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:24:25.540448 11279 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:25.569738 11279 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.68:35813
I20240502 14:24:25.569819 11406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.68:35813 every 8 connection(s)
I20240502 14:24:25.570912 11279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:24:25.580832 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 11279
I20240502 14:24:25.581043 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 14:24:25.582065 11407 heartbeater.cc:347] Connected to a master server at 127.5.107.126:40403
I20240502 14:24:25.582329 11407 heartbeater.cc:464] Registering TS with master...
I20240502 14:24:25.583024 11407 heartbeater.cc:511] Master 127.5.107.126:40403 requested a full tablet report, sending...
I20240502 14:24:25.584117 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.69:0
--local_ip_for_outbound_sockets=127.5.107.69
--webserver_interface=127.5.107.69
--webserver_port=0
--tserver_master_addrs=127.5.107.126:40403
--builtin_ntp_servers=127.5.107.84:36267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:24:25.584358 10828 ts_manager.cc:194] Registered new tserver with Master: b14f5c5b988b4542a2e1b71620e913de (127.5.107.68:35813)
I20240502 14:24:25.585494 10828 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.68:33895
I20240502 14:24:25.784024 11145 heartbeater.cc:503] Master 127.5.107.126:40403 was elected leader, sending a full tablet report...
W20240502 14:24:25.868193 11410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:24:25.868453 11410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:24:25.868566 11410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:24:25.876504 11410 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:24:25.876654 11410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:24:25.876883 11410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.69
I20240502 14:24:25.889082 11410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:36267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.5.107.69
--webserver_port=0
--tserver_master_addrs=127.5.107.126:40403
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.69
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:24:25.890327 11410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:24:25.891986 11410 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:24:25.902374 11416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:25.903020 11419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:25.903509 11417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:24:25.903739 11410 server_base.cc:1034] running on GCE node
I20240502 14:24:25.904269 11410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:24:25.905139 11410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:24:25.906335 11410 hybrid_clock.cc:648] HybridClock initialized: now 1714659865906301 us; error 26 us; skew 500 ppm
I20240502 14:24:25.906584 11410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:24:25.908623 11410 webserver.cc:460] Webserver started at http://127.5.107.69:43033/ using document root <none> and password file <none>
I20240502 14:24:25.909258 11410 fs_manager.cc:359] Metadata directory not provided
I20240502 14:24:25.909351 11410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:24:25.909615 11410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:24:25.911948 11410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "18445fc7a4d24a438cee99e8cfdb2867"
format_stamp: "Formatted at 2024-05-02 14:24:25 on dist-test-slave-2cw1"
I20240502 14:24:25.912667 11410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "18445fc7a4d24a438cee99e8cfdb2867"
format_stamp: "Formatted at 2024-05-02 14:24:25 on dist-test-slave-2cw1"
I20240502 14:24:25.916978 11410 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20240502 14:24:25.919862 11425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:25.921023 11410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:24:25.921218 11410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "18445fc7a4d24a438cee99e8cfdb2867"
format_stamp: "Formatted at 2024-05-02 14:24:25 on dist-test-slave-2cw1"
I20240502 14:24:25.921415 11410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:24:25.941767 11410 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:24:25.942459 11410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:24:25.942713 11410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:24:25.943531 11410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:24:25.944799 11410 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:24:25.944947 11410 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:25.945030 11410 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:24:25.945103 11410 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:25.972148 11410 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.69:43149
I20240502 14:24:25.972211 11537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.69:43149 every 8 connection(s)
I20240502 14:24:25.973328 11410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:24:25.979382 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 11410
I20240502 14:24:25.979522 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 14:24:25.983757 11538 heartbeater.cc:347] Connected to a master server at 127.5.107.126:40403
I20240502 14:24:25.983973 11538 heartbeater.cc:464] Registering TS with master...
I20240502 14:24:25.984519 11538 heartbeater.cc:511] Master 127.5.107.126:40403 requested a full tablet report, sending...
I20240502 14:24:25.985567 10828 ts_manager.cc:194] Registered new tserver with Master: 18445fc7a4d24a438cee99e8cfdb2867 (127.5.107.69:43149)
I20240502 14:24:25.986227 10828 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.69:46775
I20240502 14:24:25.994374 5549 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 14:24:26.007833 10828 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52066:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 14:24:26.041988 11080 tablet_service.cc:1467] Processing CreateTablet for tablet ea08b6743f3a4d5f94882d3516491679 (DEFAULT_TABLE table=TestTable [id=39aaf436834f4fe2bce233cd1f933de4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:26.043387 11080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea08b6743f3a4d5f94882d3516491679. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:26.047464 11210 tablet_service.cc:1467] Processing CreateTablet for tablet ea08b6743f3a4d5f94882d3516491679 (DEFAULT_TABLE table=TestTable [id=39aaf436834f4fe2bce233cd1f933de4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:26.048691 11210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea08b6743f3a4d5f94882d3516491679. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:26.048218 11473 tablet_service.cc:1467] Processing CreateTablet for tablet ea08b6743f3a4d5f94882d3516491679 (DEFAULT_TABLE table=TestTable [id=39aaf436834f4fe2bce233cd1f933de4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:26.049427 11473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea08b6743f3a4d5f94882d3516491679. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:26.055311 11557 tablet_bootstrap.cc:492] T ea08b6743f3a4d5f94882d3516491679 P af87759abc1044aa871c7201a9a1559a: Bootstrap starting.
I20240502 14:24:26.057957 11557 tablet_bootstrap.cc:654] T ea08b6743f3a4d5f94882d3516491679 P af87759abc1044aa871c7201a9a1559a: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:26.058844 11557 log.cc:824] T ea08b6743f3a4d5f94882d3516491679 P af87759abc1044aa871c7201a9a1559a: Log is configured to *not* fsync() on all Append() calls
I20240502 14:24:26.060593 11559 tablet_bootstrap.cc:492] T ea08b6743f3a4d5f94882d3516491679 P 766502fe87a248d78a7150f2b40c11de: Bootstrap starting.
I20240502 14:24:26.062198 11560 tablet_bootstrap.cc:492] T ea08b6743f3a4d5f94882d3516491679 P 18445fc7a4d24a438cee99e8cfdb2867: Bootstrap starting.
I20240502 14:24:26.062255 11557 tablet_bootstrap.cc:492] T ea08b6743f3a4d5f94882d3516491679 P af87759abc1044aa871c7201a9a1559a: No bootstrap required, opened a new log
I20240502 14:24:26.062544 11557 ts_tablet_manager.cc:1367] T ea08b6743f3a4d5f94882d3516491679 P af87759abc1044aa871c7201a9a1559a: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240502 14:24:26.063938 11559 tablet_bootstrap.cc:654] T ea08b6743f3a4d5f94882d3516491679 P 766502fe87a248d78a7150f2b40c11de: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:26.064662 11560 tablet_bootstrap.cc:654] T ea08b6743f3a4d5f94882d3516491679 P 18445fc7a4d24a438cee99e8cfdb2867: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:26.064977 11559 log.cc:824] T ea08b6743f3a4d5f94882d3516491679 P 766502fe87a248d78a7150f2b40c11de: Log is configured to *not* fsync() on all Append() calls
I20240502 14:24:26.065588 11560 log.cc:824] T ea08b6743f3a4d5f94882d3516491679 P 18445fc7a4d24a438cee99e8cfdb2867: Log is configured to *not* fsync() on all Append() calls
I20240502 14:24:26.067330 11559 tablet_bootstrap.cc:492] T ea08b6743f3a4d5f94882d3516491679 P 766502fe87a248d78a7150f2b40c11de: No bootstrap required, opened a new log
I20240502 14:24:26.067572 11559 ts_tablet_manager.cc:1367] T ea08b6743f3a4d5f94882d3516491679 P 766502fe87a248d78a7150f2b40c11de: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 14:24:26.067636 11560 tablet_bootstrap.cc:492] T ea08b6743f3a4d5f94882d3516491679 P 18445fc7a4d24a438cee99e8cfdb2867: No bootstrap required, opened a new log
I20240502 14:24:26.067836 11560 ts_tablet_manager.cc:1367] T ea08b6743f3a4d5f94882d3516491679 P 18445fc7a4d24a438cee99e8cfdb2867: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240502 14:24:26.070466 11557 raft_consensus.cc:348] T ea08b6743f3a4d5f94882d3516491679 P af87759abc1044aa871c7201a9a1559a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af87759abc1044aa871c7201a9a1559a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 39413 } } peers { permanent_uuid: "18445fc7a4d24a438cee99e8cfdb2867" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 43149 } } peers { permanent_uuid: "766502fe87a248d78a7150f2b40c11de" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 32873 } }
I20240502 14:24:26.070879 11557 raft_consensus.cc:374] T ea08b6743f3a4d5f94882d3516491679 P af87759abc1044aa871c7201a9a1559a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:26.071043 11557 raft_consensus.cc:729] T ea08b6743f3a4d5f94882d3516491679 P af87759abc1044aa871c7201a9a1559a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af87759abc1044aa871c7201a9a1559a, State: Initialized, Role: FOLLOWER
I20240502 14:24:26.072160 11557 consensus_queue.cc:260] T ea08b6743f3a4d5f94882d3516491679 P af87759abc1044aa871c7201a9a1559a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af87759abc1044aa871c7201a9a1559a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 39413 } } peers { permanent_uuid: "18445fc7a4d24a438cee99e8cfdb2867" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 43149 } } peers { permanent_uuid: "766502fe87a248d78a7150f2b40c11de" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 32873 } }
I20240502 14:24:26.073923 11557 ts_tablet_manager.cc:1398] T ea08b6743f3a4d5f94882d3516491679 P af87759abc1044aa871c7201a9a1559a: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20240502 14:24:26.073922 11560 raft_consensus.cc:348] T ea08b6743f3a4d5f94882d3516491679 P 18445fc7a4d24a438cee99e8cfdb2867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af87759abc1044aa871c7201a9a1559a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 39413 } } peers { permanent_uuid: "18445fc7a4d24a438cee99e8cfdb2867" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 43149 } } peers { permanent_uuid: "766502fe87a248d78a7150f2b40c11de" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 32873 } }
I20240502 14:24:26.074349 11560 raft_consensus.cc:374] T ea08b6743f3a4d5f94882d3516491679 P 18445fc7a4d24a438cee99e8cfdb2867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:26.074461 11560 raft_consensus.cc:729] T ea08b6743f3a4d5f94882d3516491679 P 18445fc7a4d24a438cee99e8cfdb2867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18445fc7a4d24a438cee99e8cfdb2867, State: Initialized, Role: FOLLOWER
I20240502 14:24:26.075052 11560 consensus_queue.cc:260] T ea08b6743f3a4d5f94882d3516491679 P 18445fc7a4d24a438cee99e8cfdb2867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af87759abc1044aa871c7201a9a1559a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 39413 } } peers { permanent_uuid: "18445fc7a4d24a438cee99e8cfdb2867" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 43149 } } peers { permanent_uuid: "766502fe87a248d78a7150f2b40c11de" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 32873 } }
I20240502 14:24:26.075973 11538 heartbeater.cc:503] Master 127.5.107.126:40403 was elected leader, sending a full tablet report...
I20240502 14:24:26.075999 11559 raft_consensus.cc:348] T ea08b6743f3a4d5f94882d3516491679 P 766502fe87a248d78a7150f2b40c11de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af87759abc1044aa871c7201a9a1559a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 39413 } } peers { permanent_uuid: "18445fc7a4d24a438cee99e8cfdb2867" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 43149 } } peers { permanent_uuid: "766502fe87a248d78a7150f2b40c11de" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 32873 } }
I20240502 14:24:26.076375 11559 raft_consensus.cc:374] T ea08b6743f3a4d5f94882d3516491679 P 766502fe87a248d78a7150f2b40c11de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:26.076457 11559 raft_consensus.cc:729] T ea08b6743f3a4d5f94882d3516491679 P 766502fe87a248d78a7150f2b40c11de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 766502fe87a248d78a7150f2b40c11de, State: Initialized, Role: FOLLOWER
I20240502 14:24:26.076498 11560 ts_tablet_manager.cc:1398] T ea08b6743f3a4d5f94882d3516491679 P 18445fc7a4d24a438cee99e8cfdb2867: Time spent starting tablet: real 0.009s user 0.004s sys 0.005s
I20240502 14:24:26.077145 11559 consensus_queue.cc:260] T ea08b6743f3a4d5f94882d3516491679 P 766502fe87a248d78a7150f2b40c11de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af87759abc1044aa871c7201a9a1559a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 39413 } } peers { permanent_uuid: "18445fc7a4d24a438cee99e8cfdb2867" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 43149 } } peers { permanent_uuid: "766502fe87a248d78a7150f2b40c11de" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 32873 } }
I20240502 14:24:26.078111 11276 heartbeater.cc:503] Master 127.5.107.126:40403 was elected leader, sending a full tablet report...
I20240502 14:24:26.078564 11559 ts_tablet_manager.cc:1398] T ea08b6743f3a4d5f94882d3516491679 P 766502fe87a248d78a7150f2b40c11de: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20240502 14:24:26.134552 11564 raft_consensus.cc:482] T ea08b6743f3a4d5f94882d3516491679 P 18445fc7a4d24a438cee99e8cfdb2867 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:24:26.134930 11564 raft_consensus.cc:504] T ea08b6743f3a4d5f94882d3516491679 P 18445fc7a4d24a438cee99e8cfdb2867 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af87759abc1044aa871c7201a9a1559a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 39413 } } peers { permanent_uuid: "18445fc7a4d24a438cee99e8cfdb2867" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 43149 } } peers { permanent_uuid: "766502fe87a248d78a7150f2b40c11de" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 32873 } }
I20240502 14:24:26.136461 11564 leader_election.cc:288] T ea08b6743f3a4d5f94882d3516491679 P 18445fc7a4d24a438cee99e8cfdb2867 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af87759abc1044aa871c7201a9a1559a (127.5.107.66:39413), 766502fe87a248d78a7150f2b40c11de (127.5.107.67:32873)
I20240502 14:24:26.141829 11100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea08b6743f3a4d5f94882d3516491679" candidate_uuid: "18445fc7a4d24a438cee99e8cfdb2867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af87759abc1044aa871c7201a9a1559a" is_pre_election: true
I20240502 14:24:26.142285 11100 raft_consensus.cc:2454] T ea08b6743f3a4d5f94882d3516491679 P af87759abc1044aa871c7201a9a1559a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18445fc7a4d24a438cee99e8cfdb2867 in term 0.
I20240502 14:24:26.142948 11426 leader_election.cc:302] T ea08b6743f3a4d5f94882d3516491679 P 18445fc7a4d24a438cee99e8cfdb2867 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18445fc7a4d24a438cee99e8cfdb2867, af87759abc1044aa871c7201a9a1559a; no voters:
I20240502 14:24:26.143040 11230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea08b6743f3a4d5f94882d3516491679" candidate_uuid: "18445fc7a4d24a438cee99e8cfdb2867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "766502fe87a248d78a7150f2b40c11de" is_pre_election: true
I20240502 14:24:26.143322 11564 raft_consensus.cc:2781] T ea08b6743f3a4d5f94882d3516491679 P 18445fc7a4d24a438cee99e8cfdb2867 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:24:26.143458 11564 raft_consensus.cc:482] T ea08b6743f3a4d5f94882d3516491679 P 18445fc7a4d24a438cee99e8cfdb2867 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:24:26.143471 11230 raft_consensus.cc:2454] T ea08b6743f3a4d5f94882d3516491679 P 766502fe87a248d78a7150f2b40c11de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18445fc7a4d24a438cee99e8cfdb2867 in term 0.
I20240502 14:24:26.143549 11564 raft_consensus.cc:3037] T ea08b6743f3a4d5f94882d3516491679 P 18445fc7a4d24a438cee99e8cfdb2867 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:26.145555 11564 raft_consensus.cc:504] T ea08b6743f3a4d5f94882d3516491679 P 18445fc7a4d24a438cee99e8cfdb2867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af87759abc1044aa871c7201a9a1559a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 39413 } } peers { permanent_uuid: "18445fc7a4d24a438cee99e8cfdb2867" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 43149 } } peers { permanent_uuid: "766502fe87a248d78a7150f2b40c11de" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 32873 } }
I20240502 14:24:26.146059 11564 leader_election.cc:288] T ea08b6743f3a4d5f94882d3516491679 P 18445fc7a4d24a438cee99e8cfdb2867 [CANDIDATE]: Term 1 election: Requested vote from peers af87759abc1044aa871c7201a9a1559a (127.5.107.66:39413), 766502fe87a248d78a7150f2b40c11de (127.5.107.67:32873)
I20240502 14:24:26.146392 11100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea08b6743f3a4d5f94882d3516491679" candidate_uuid: "18445fc7a4d24a438cee99e8cfdb2867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af87759abc1044aa871c7201a9a1559a"
I20240502 14:24:26.146531 11100 raft_consensus.cc:3037] T ea08b6743f3a4d5f94882d3516491679 P af87759abc1044aa871c7201a9a1559a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:26.146518 11230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea08b6743f3a4d5f94882d3516491679" candidate_uuid: "18445fc7a4d24a438cee99e8cfdb2867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "766502fe87a248d78a7150f2b40c11de"
I20240502 14:24:26.146668 11230 raft_consensus.cc:3037] T ea08b6743f3a4d5f94882d3516491679 P 766502fe87a248d78a7150f2b40c11de [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:26.148325 11100 raft_consensus.cc:2454] T ea08b6743f3a4d5f94882d3516491679 P af87759abc1044aa871c7201a9a1559a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18445fc7a4d24a438cee99e8cfdb2867 in term 1.
I20240502 14:24:26.148546 11230 raft_consensus.cc:2454] T ea08b6743f3a4d5f94882d3516491679 P 766502fe87a248d78a7150f2b40c11de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18445fc7a4d24a438cee99e8cfdb2867 in term 1.
I20240502 14:24:26.148653 11426 leader_election.cc:302] T ea08b6743f3a4d5f94882d3516491679 P 18445fc7a4d24a438cee99e8cfdb2867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18445fc7a4d24a438cee99e8cfdb2867, af87759abc1044aa871c7201a9a1559a; no voters:
I20240502 14:24:26.148903 11564 raft_consensus.cc:2781] T ea08b6743f3a4d5f94882d3516491679 P 18445fc7a4d24a438cee99e8cfdb2867 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:24:26.149312 11564 raft_consensus.cc:686] T ea08b6743f3a4d5f94882d3516491679 P 18445fc7a4d24a438cee99e8cfdb2867 [term 1 LEADER]: Becoming Leader. State: Replica: 18445fc7a4d24a438cee99e8cfdb2867, State: Running, Role: LEADER
I20240502 14:24:26.149780 11564 consensus_queue.cc:237] T ea08b6743f3a4d5f94882d3516491679 P 18445fc7a4d24a438cee99e8cfdb2867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af87759abc1044aa871c7201a9a1559a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 39413 } } peers { permanent_uuid: "18445fc7a4d24a438cee99e8cfdb2867" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 43149 } } peers { permanent_uuid: "766502fe87a248d78a7150f2b40c11de" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 32873 } }
I20240502 14:24:26.153571 10829 catalog_manager.cc:5484] T ea08b6743f3a4d5f94882d3516491679 P 18445fc7a4d24a438cee99e8cfdb2867 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18445fc7a4d24a438cee99e8cfdb2867 (127.5.107.69). New cstate: current_term: 1 leader_uuid: "18445fc7a4d24a438cee99e8cfdb2867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af87759abc1044aa871c7201a9a1559a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 39413 } } peers { permanent_uuid: "18445fc7a4d24a438cee99e8cfdb2867" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 43149 } } peers { permanent_uuid: "766502fe87a248d78a7150f2b40c11de" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 32873 } } }
W20240502 14:24:26.157685 11277 tablet.cc:2369] T ea08b6743f3a4d5f94882d3516491679 P 766502fe87a248d78a7150f2b40c11de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:24:26.178622 5549 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 14:24:26.180968 5549 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 60fa049c8dc94ca7935b892be97a7f80 to finish bootstrapping
I20240502 14:24:26.188740 5549 ts_itest-base.cc:233] Waiting for 1 tablets on tserver af87759abc1044aa871c7201a9a1559a to finish bootstrapping
I20240502 14:24:26.195341 5549 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 766502fe87a248d78a7150f2b40c11de to finish bootstrapping
I20240502 14:24:26.201403 5549 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b14f5c5b988b4542a2e1b71620e913de to finish bootstrapping
I20240502 14:24:26.208271 5549 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 18445fc7a4d24a438cee99e8cfdb2867 to finish bootstrapping
W20240502 14:24:26.224715 11539 tablet.cc:2369] T ea08b6743f3a4d5f94882d3516491679 P 18445fc7a4d24a438cee99e8cfdb2867: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:24:26.270179 11146 tablet.cc:2369] T ea08b6743f3a4d5f94882d3516491679 P af87759abc1044aa871c7201a9a1559a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:24:26.587600 11407 heartbeater.cc:503] Master 127.5.107.126:40403 was elected leader, sending a full tablet report...
I20240502 14:24:26.620188 11473 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:24:26.620755 11564 consensus_queue.cc:1035] T ea08b6743f3a4d5f94882d3516491679 P 18445fc7a4d24a438cee99e8cfdb2867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "766502fe87a248d78a7150f2b40c11de" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 32873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:24:26.620967 10950 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:26.631981 11080 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:26.652127 11210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:26.657502 11567 consensus_queue.cc:1035] T ea08b6743f3a4d5f94882d3516491679 P 18445fc7a4d24a438cee99e8cfdb2867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af87759abc1044aa871c7201a9a1559a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:24:26.669032 11342 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:26.800985 11473 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:24:26.803345 10950 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:26.807723 11210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:26.813941 11080 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:26.815145 11342 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:27.394850 10950 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:27.401502 11473 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:24:27.410450 11080 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:27.410792 11342 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:27.411145 11210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:27.549077 11473 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:24:27.558199 10950 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:27.560987 11342 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:27.560992 11210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:27.569752 11080 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:27.707115 11473 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:24:27.708662 10950 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:27.718694 11210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:27.719738 11080 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:27.726374 11342 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:27.866758 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 10888
I20240502 14:24:27.868218 11009 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:24:27.949712 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 10888
I20240502 14:24:27.967037 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 11018
I20240502 14:24:27.968081 11140 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:24:28.057390 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 11018
I20240502 14:24:28.074926 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 11148
I20240502 14:24:28.075776 11271 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:24:28.160058 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 11148
I20240502 14:24:28.176577 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 11279
I20240502 14:24:28.177527 11401 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:24:28.257073 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 11279
W20240502 14:24:28.261664 11426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.67:32873: connect: Connection refused (error 111)
W20240502 14:24:28.263160 11426 consensus_peers.cc:485] T ea08b6743f3a4d5f94882d3516491679 P 18445fc7a4d24a438cee99e8cfdb2867 -> Peer 766502fe87a248d78a7150f2b40c11de (127.5.107.67:32873): Couldn't send request to peer 766502fe87a248d78a7150f2b40c11de. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.67:32873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:24:28.273908 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 11410
I20240502 14:24:28.274691 11533 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:24:28.359491 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 11410
I20240502 14:24:28.375968 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 10798
I20240502 14:24:28.376889 10859 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:24:28.480746 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 10798
2024-05-02T14:24:28Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0 (5048 ms)
[----------] 1 test from RebalanceStartCriteriaTest (5048 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/2
I20240502 14:24:28.520337 5549 test_util.cc:268] Using random seed: 1549774719
I20240502 14:24:28.522111 5549 ts_itest-base.cc:112] Starting cluster with:
I20240502 14:24:28.522176 5549 ts_itest-base.cc:113] --------------
I20240502 14:24:28.522220 5549 ts_itest-base.cc:114] 7 tablet servers
I20240502 14:24:28.522258 5549 ts_itest-base.cc:115] 2 replicas per TS
I20240502 14:24:28.522307 5549 ts_itest-base.cc:116] --------------
2024-05-02T14:24:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:24:28Z Disabled control of system clock
I20240502 14:24:28.544081 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.107.126:36087
--webserver_interface=127.5.107.126
--webserver_port=0
--builtin_ntp_servers=127.5.107.84:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.107.126:36087
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240502 14:24:28.832821 11762 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 14:24:28.833117 11762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:24:28.833197 11762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:24:28.833279 11762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:24:28.839808 11762 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:24:28.839895 11762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:24:28.839948 11762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:24:28.839987 11762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:24:28.840044 11762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:24:28.848183 11762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.5.107.126:36087
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.107.126:36087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.107.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:24:28.849244 11762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:24:28.850773 11762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:24:28.860438 11767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:28.861140 11768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:28.861665 11770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:24:28.862316 11762 server_base.cc:1034] running on GCE node
I20240502 14:24:28.862989 11762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:24:28.864183 11762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:24:28.865412 11762 hybrid_clock.cc:648] HybridClock initialized: now 1714659868865335 us; error 37 us; skew 500 ppm
I20240502 14:24:28.865736 11762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:24:28.871436 11762 webserver.cc:460] Webserver started at http://127.5.107.126:34443/ using document root <none> and password file <none>
I20240502 14:24:28.872069 11762 fs_manager.cc:359] Metadata directory not provided
I20240502 14:24:28.872190 11762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:24:28.872463 11762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:24:28.874868 11762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "525d10aecf2c4fbb9c5cdfe9286d0a80"
format_stamp: "Formatted at 2024-05-02 14:24:28 on dist-test-slave-2cw1"
I20240502 14:24:28.875526 11762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "525d10aecf2c4fbb9c5cdfe9286d0a80"
format_stamp: "Formatted at 2024-05-02 14:24:28 on dist-test-slave-2cw1"
I20240502 14:24:28.879837 11762 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 14:24:28.882784 11776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:28.884032 11762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 14:24:28.884212 11762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/wal
uuid: "525d10aecf2c4fbb9c5cdfe9286d0a80"
format_stamp: "Formatted at 2024-05-02 14:24:28 on dist-test-slave-2cw1"
I20240502 14:24:28.884407 11762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:24:28.905089 11762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:24:28.905833 11762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:24:28.906111 11762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:24:28.922869 11762 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.126:36087
I20240502 14:24:28.922924 11827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.126:36087 every 8 connection(s)
I20240502 14:24:28.924064 11762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 14:24:28.926710 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 11762
I20240502 14:24:28.926898 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 14:24:28.927155 11828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:28.937942 11828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 525d10aecf2c4fbb9c5cdfe9286d0a80: Bootstrap starting.
I20240502 14:24:28.940363 11828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 525d10aecf2c4fbb9c5cdfe9286d0a80: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:28.941226 11828 log.cc:824] T 00000000000000000000000000000000 P 525d10aecf2c4fbb9c5cdfe9286d0a80: Log is configured to *not* fsync() on all Append() calls
I20240502 14:24:28.943346 11828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 525d10aecf2c4fbb9c5cdfe9286d0a80: No bootstrap required, opened a new log
I20240502 14:24:28.948035 11828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 525d10aecf2c4fbb9c5cdfe9286d0a80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525d10aecf2c4fbb9c5cdfe9286d0a80" member_type: VOTER last_known_addr { host: "127.5.107.126" port: 36087 } }
I20240502 14:24:28.948293 11828 raft_consensus.cc:374] T 00000000000000000000000000000000 P 525d10aecf2c4fbb9c5cdfe9286d0a80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:28.948369 11828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 525d10aecf2c4fbb9c5cdfe9286d0a80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525d10aecf2c4fbb9c5cdfe9286d0a80, State: Initialized, Role: FOLLOWER
I20240502 14:24:28.948824 11828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 525d10aecf2c4fbb9c5cdfe9286d0a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525d10aecf2c4fbb9c5cdfe9286d0a80" member_type: VOTER last_known_addr { host: "127.5.107.126" port: 36087 } }
I20240502 14:24:28.948993 11828 raft_consensus.cc:388] T 00000000000000000000000000000000 P 525d10aecf2c4fbb9c5cdfe9286d0a80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:24:28.949105 11828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 525d10aecf2c4fbb9c5cdfe9286d0a80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:24:28.949249 11828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 525d10aecf2c4fbb9c5cdfe9286d0a80 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:28.950872 11828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 525d10aecf2c4fbb9c5cdfe9286d0a80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525d10aecf2c4fbb9c5cdfe9286d0a80" member_type: VOTER last_known_addr { host: "127.5.107.126" port: 36087 } }
I20240502 14:24:28.951243 11828 leader_election.cc:302] T 00000000000000000000000000000000 P 525d10aecf2c4fbb9c5cdfe9286d0a80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 525d10aecf2c4fbb9c5cdfe9286d0a80; no voters:
I20240502 14:24:28.951671 11828 leader_election.cc:288] T 00000000000000000000000000000000 P 525d10aecf2c4fbb9c5cdfe9286d0a80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:24:28.951804 11833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 525d10aecf2c4fbb9c5cdfe9286d0a80 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:24:28.952416 11833 raft_consensus.cc:686] T 00000000000000000000000000000000 P 525d10aecf2c4fbb9c5cdfe9286d0a80 [term 1 LEADER]: Becoming Leader. State: Replica: 525d10aecf2c4fbb9c5cdfe9286d0a80, State: Running, Role: LEADER
I20240502 14:24:28.952945 11833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 525d10aecf2c4fbb9c5cdfe9286d0a80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525d10aecf2c4fbb9c5cdfe9286d0a80" member_type: VOTER last_known_addr { host: "127.5.107.126" port: 36087 } }
I20240502 14:24:28.953370 11828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 525d10aecf2c4fbb9c5cdfe9286d0a80 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:24:28.956037 11835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 525d10aecf2c4fbb9c5cdfe9286d0a80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 525d10aecf2c4fbb9c5cdfe9286d0a80. Latest consensus state: current_term: 1 leader_uuid: "525d10aecf2c4fbb9c5cdfe9286d0a80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525d10aecf2c4fbb9c5cdfe9286d0a80" member_type: VOTER last_known_addr { host: "127.5.107.126" port: 36087 } } }
I20240502 14:24:28.956351 11835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 525d10aecf2c4fbb9c5cdfe9286d0a80 [sys.catalog]: This master's current role is: LEADER
I20240502 14:24:28.957544 11834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 525d10aecf2c4fbb9c5cdfe9286d0a80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "525d10aecf2c4fbb9c5cdfe9286d0a80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525d10aecf2c4fbb9c5cdfe9286d0a80" member_type: VOTER last_known_addr { host: "127.5.107.126" port: 36087 } } }
I20240502 14:24:28.957778 11843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:24:28.957845 11834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 525d10aecf2c4fbb9c5cdfe9286d0a80 [sys.catalog]: This master's current role is: LEADER
I20240502 14:24:28.962412 11843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:24:28.969892 11843 catalog_manager.cc:1347] Generated new cluster ID: f96f7d345c5d4b03a5c6c370b42b1d99
I20240502 14:24:28.969988 11843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:24:28.987222 11843 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:24:28.988140 11843 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:24:28.996162 11843 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 525d10aecf2c4fbb9c5cdfe9286d0a80: Generated new TSK 0
I20240502 14:24:28.996701 11843 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:24:29.001690 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.65:0
--local_ip_for_outbound_sockets=127.5.107.65
--webserver_interface=127.5.107.65
--webserver_port=0
--tserver_master_addrs=127.5.107.126:36087
--builtin_ntp_servers=127.5.107.84:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 14:24:29.315861 11853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:24:29.316088 11853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:24:29.316182 11853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:24:29.322849 11853 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:24:29.322959 11853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:24:29.323115 11853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.65
I20240502 14:24:29.331030 11853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.107.65
--webserver_port=0
--tserver_master_addrs=127.5.107.126:36087
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.65
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:24:29.332170 11853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:24:29.333809 11853 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:24:29.344903 11861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:29.345546 11859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:29.345657 11858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:24:29.346261 11853 server_base.cc:1034] running on GCE node
I20240502 14:24:29.346750 11853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:24:29.347507 11853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:24:29.348685 11853 hybrid_clock.cc:648] HybridClock initialized: now 1714659869348650 us; error 30 us; skew 500 ppm
I20240502 14:24:29.348942 11853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:24:29.350809 11853 webserver.cc:460] Webserver started at http://127.5.107.65:37089/ using document root <none> and password file <none>
I20240502 14:24:29.351428 11853 fs_manager.cc:359] Metadata directory not provided
I20240502 14:24:29.351533 11853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:24:29.351840 11853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:24:29.355736 11853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "0e41abe35d8f40a2a01953eea1c5d092"
format_stamp: "Formatted at 2024-05-02 14:24:29 on dist-test-slave-2cw1"
I20240502 14:24:29.356391 11853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "0e41abe35d8f40a2a01953eea1c5d092"
format_stamp: "Formatted at 2024-05-02 14:24:29 on dist-test-slave-2cw1"
I20240502 14:24:29.360868 11853 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 14:24:29.363955 11867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:29.365010 11853 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:24:29.365135 11853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "0e41abe35d8f40a2a01953eea1c5d092"
format_stamp: "Formatted at 2024-05-02 14:24:29 on dist-test-slave-2cw1"
I20240502 14:24:29.365336 11853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:24:29.378521 11853 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:24:29.379176 11853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:24:29.379400 11853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:24:29.380169 11853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:24:29.381409 11853 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:24:29.381500 11853 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:29.381606 11853 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:24:29.381676 11853 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:29.409008 11853 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.65:36871
I20240502 14:24:29.409076 11980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.65:36871 every 8 connection(s)
I20240502 14:24:29.410116 11853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 14:24:29.419790 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 11853
I20240502 14:24:29.420049 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 14:24:29.420641 11981 heartbeater.cc:347] Connected to a master server at 127.5.107.126:36087
I20240502 14:24:29.420909 11981 heartbeater.cc:464] Registering TS with master...
I20240502 14:24:29.421698 11981 heartbeater.cc:511] Master 127.5.107.126:36087 requested a full tablet report, sending...
I20240502 14:24:29.423218 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.66:0
--local_ip_for_outbound_sockets=127.5.107.66
--webserver_interface=127.5.107.66
--webserver_port=0
--tserver_master_addrs=127.5.107.126:36087
--builtin_ntp_servers=127.5.107.84:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:24:29.423532 11793 ts_manager.cc:194] Registered new tserver with Master: 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871)
I20240502 14:24:29.424816 11793 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.65:58643
W20240502 14:24:29.711325 11984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:24:29.711583 11984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:24:29.711706 11984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:24:29.718276 11984 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:24:29.718389 11984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:24:29.718520 11984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.66
I20240502 14:24:29.726354 11984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.107.66
--webserver_port=0
--tserver_master_addrs=127.5.107.126:36087
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.66
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:24:29.727432 11984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:24:29.728955 11984 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:24:29.739249 11989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:29.739995 11990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:29.740644 11992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:24:29.740782 11984 server_base.cc:1034] running on GCE node
I20240502 14:24:29.741344 11984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:24:29.742159 11984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:24:29.743335 11984 hybrid_clock.cc:648] HybridClock initialized: now 1714659869743298 us; error 33 us; skew 500 ppm
I20240502 14:24:29.743588 11984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:24:29.745389 11984 webserver.cc:460] Webserver started at http://127.5.107.66:35351/ using document root <none> and password file <none>
I20240502 14:24:29.745961 11984 fs_manager.cc:359] Metadata directory not provided
I20240502 14:24:29.746042 11984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:24:29.746291 11984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:24:29.748327 11984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "36e66c7b1a03458b89a91c56a57df72a"
format_stamp: "Formatted at 2024-05-02 14:24:29 on dist-test-slave-2cw1"
I20240502 14:24:29.748968 11984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "36e66c7b1a03458b89a91c56a57df72a"
format_stamp: "Formatted at 2024-05-02 14:24:29 on dist-test-slave-2cw1"
I20240502 14:24:29.753263 11984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 14:24:29.755945 11998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:29.756942 11984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:24:29.757074 11984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "36e66c7b1a03458b89a91c56a57df72a"
format_stamp: "Formatted at 2024-05-02 14:24:29 on dist-test-slave-2cw1"
I20240502 14:24:29.757274 11984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:24:29.774678 11984 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:24:29.775274 11984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:24:29.775481 11984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:24:29.776206 11984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:24:29.777395 11984 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:24:29.777469 11984 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:29.777565 11984 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:24:29.777622 11984 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:29.804564 11984 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.66:35445
I20240502 14:24:29.804610 12112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.66:35445 every 8 connection(s)
I20240502 14:24:29.805598 11984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 14:24:29.808724 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 11984
I20240502 14:24:29.808899 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 14:24:29.811570 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.67:0
--local_ip_for_outbound_sockets=127.5.107.67
--webserver_interface=127.5.107.67
--webserver_port=0
--tserver_master_addrs=127.5.107.126:36087
--builtin_ntp_servers=127.5.107.84:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:24:29.816375 12113 heartbeater.cc:347] Connected to a master server at 127.5.107.126:36087
I20240502 14:24:29.816591 12113 heartbeater.cc:464] Registering TS with master...
I20240502 14:24:29.817131 12113 heartbeater.cc:511] Master 127.5.107.126:36087 requested a full tablet report, sending...
I20240502 14:24:29.818323 11793 ts_manager.cc:194] Registered new tserver with Master: 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445)
I20240502 14:24:29.819299 11793 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.66:37535
W20240502 14:24:30.095659 12116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:24:30.095888 12116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:24:30.095968 12116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:24:30.102093 12116 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:24:30.102188 12116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:24:30.102295 12116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.67
I20240502 14:24:30.109594 12116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.107.67
--webserver_port=0
--tserver_master_addrs=127.5.107.126:36087
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.67
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:24:30.110522 12116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:24:30.111951 12116 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:24:30.122047 12125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:30.122730 12123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:30.123440 12121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:24:30.124012 12116 server_base.cc:1034] running on GCE node
I20240502 14:24:30.124495 12116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:24:30.125334 12116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:24:30.126555 12116 hybrid_clock.cc:648] HybridClock initialized: now 1714659870126502 us; error 47 us; skew 500 ppm
I20240502 14:24:30.126863 12116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:24:30.128785 12116 webserver.cc:460] Webserver started at http://127.5.107.67:44207/ using document root <none> and password file <none>
I20240502 14:24:30.129455 12116 fs_manager.cc:359] Metadata directory not provided
I20240502 14:24:30.129577 12116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:24:30.129874 12116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:24:30.132213 12116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "d9748c575342401bb80683c7e2c4e105"
format_stamp: "Formatted at 2024-05-02 14:24:30 on dist-test-slave-2cw1"
I20240502 14:24:30.133003 12116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "d9748c575342401bb80683c7e2c4e105"
format_stamp: "Formatted at 2024-05-02 14:24:30 on dist-test-slave-2cw1"
I20240502 14:24:30.137655 12116 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 14:24:30.140424 12131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:30.141602 12116 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:24:30.141755 12116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d9748c575342401bb80683c7e2c4e105"
format_stamp: "Formatted at 2024-05-02 14:24:30 on dist-test-slave-2cw1"
I20240502 14:24:30.141947 12116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:24:30.151566 12116 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:24:30.152253 12116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:24:30.152500 12116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:24:30.153362 12116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:24:30.154570 12116 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:24:30.154677 12116 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:30.154771 12116 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:24:30.154843 12116 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:30.182320 12116 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.67:45923
I20240502 14:24:30.182384 12244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.67:45923 every 8 connection(s)
I20240502 14:24:30.183395 12116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 14:24:30.188100 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 12116
I20240502 14:24:30.188261 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 14:24:30.190941 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.68:0
--local_ip_for_outbound_sockets=127.5.107.68
--webserver_interface=127.5.107.68
--webserver_port=0
--tserver_master_addrs=127.5.107.126:36087
--builtin_ntp_servers=127.5.107.84:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:24:30.194182 12245 heartbeater.cc:347] Connected to a master server at 127.5.107.126:36087
I20240502 14:24:30.194398 12245 heartbeater.cc:464] Registering TS with master...
I20240502 14:24:30.194938 12245 heartbeater.cc:511] Master 127.5.107.126:36087 requested a full tablet report, sending...
I20240502 14:24:30.196012 11793 ts_manager.cc:194] Registered new tserver with Master: d9748c575342401bb80683c7e2c4e105 (127.5.107.67:45923)
I20240502 14:24:30.196650 11793 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.67:51617
I20240502 14:24:30.427035 11981 heartbeater.cc:503] Master 127.5.107.126:36087 was elected leader, sending a full tablet report...
W20240502 14:24:30.474817 12248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:24:30.475119 12248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:24:30.475260 12248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:24:30.482218 12248 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:24:30.482344 12248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:24:30.482496 12248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.68
I20240502 14:24:30.490149 12248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.5.107.68
--webserver_port=0
--tserver_master_addrs=127.5.107.126:36087
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.68
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:24:30.491211 12248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:24:30.492826 12248 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:24:30.503547 12254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:30.504197 12253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:30.504761 12256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:24:30.505352 12248 server_base.cc:1034] running on GCE node
I20240502 14:24:30.505818 12248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:24:30.506575 12248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:24:30.507772 12248 hybrid_clock.cc:648] HybridClock initialized: now 1714659870507734 us; error 34 us; skew 500 ppm
I20240502 14:24:30.508024 12248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:24:30.509847 12248 webserver.cc:460] Webserver started at http://127.5.107.68:36281/ using document root <none> and password file <none>
I20240502 14:24:30.510424 12248 fs_manager.cc:359] Metadata directory not provided
I20240502 14:24:30.510519 12248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:24:30.510777 12248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:24:30.512974 12248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "93c82e87d8af478292ffe1a3d3986049"
format_stamp: "Formatted at 2024-05-02 14:24:30 on dist-test-slave-2cw1"
I20240502 14:24:30.513677 12248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "93c82e87d8af478292ffe1a3d3986049"
format_stamp: "Formatted at 2024-05-02 14:24:30 on dist-test-slave-2cw1"
I20240502 14:24:30.517912 12248 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240502 14:24:30.521067 12262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:30.522439 12248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 14:24:30.522580 12248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "93c82e87d8af478292ffe1a3d3986049"
format_stamp: "Formatted at 2024-05-02 14:24:30 on dist-test-slave-2cw1"
I20240502 14:24:30.522755 12248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:24:30.551935 12248 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:24:30.552611 12248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:24:30.552855 12248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:24:30.553768 12248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:24:30.555037 12248 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:24:30.555120 12248 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:30.555210 12248 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:24:30.555270 12248 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:30.583550 12248 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.68:37273
I20240502 14:24:30.583607 12375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.68:37273 every 8 connection(s)
I20240502 14:24:30.584671 12248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:24:30.586589 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 12248
I20240502 14:24:30.586797 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 14:24:30.590046 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.69:0
--local_ip_for_outbound_sockets=127.5.107.69
--webserver_interface=127.5.107.69
--webserver_port=0
--tserver_master_addrs=127.5.107.126:36087
--builtin_ntp_servers=127.5.107.84:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:24:30.597076 12376 heartbeater.cc:347] Connected to a master server at 127.5.107.126:36087
I20240502 14:24:30.597357 12376 heartbeater.cc:464] Registering TS with master...
I20240502 14:24:30.597944 12376 heartbeater.cc:511] Master 127.5.107.126:36087 requested a full tablet report, sending...
I20240502 14:24:30.599090 11793 ts_manager.cc:194] Registered new tserver with Master: 93c82e87d8af478292ffe1a3d3986049 (127.5.107.68:37273)
I20240502 14:24:30.599807 11793 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.68:56365
I20240502 14:24:30.821360 12113 heartbeater.cc:503] Master 127.5.107.126:36087 was elected leader, sending a full tablet report...
W20240502 14:24:30.874344 12379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:24:30.874572 12379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:24:30.874665 12379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:24:30.880911 12379 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:24:30.881014 12379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:24:30.881132 12379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.69
I20240502 14:24:30.888906 12379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.5.107.69
--webserver_port=0
--tserver_master_addrs=127.5.107.126:36087
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.69
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:24:30.889969 12379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:24:30.891399 12379 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:24:30.902088 12384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:30.902688 12385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:30.903216 12387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:24:30.903488 12379 server_base.cc:1034] running on GCE node
I20240502 14:24:30.903942 12379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:24:30.904636 12379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:24:30.905812 12379 hybrid_clock.cc:648] HybridClock initialized: now 1714659870905776 us; error 25 us; skew 500 ppm
I20240502 14:24:30.906124 12379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:24:30.908149 12379 webserver.cc:460] Webserver started at http://127.5.107.69:33211/ using document root <none> and password file <none>
I20240502 14:24:30.908747 12379 fs_manager.cc:359] Metadata directory not provided
I20240502 14:24:30.908844 12379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:24:30.909111 12379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:24:30.911433 12379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "723fb53f34c94c6aa5f7ee3a65711268"
format_stamp: "Formatted at 2024-05-02 14:24:30 on dist-test-slave-2cw1"
I20240502 14:24:30.912137 12379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "723fb53f34c94c6aa5f7ee3a65711268"
format_stamp: "Formatted at 2024-05-02 14:24:30 on dist-test-slave-2cw1"
I20240502 14:24:30.916394 12379 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 14:24:30.919354 12393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:30.920439 12379 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:24:30.920560 12379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "723fb53f34c94c6aa5f7ee3a65711268"
format_stamp: "Formatted at 2024-05-02 14:24:30 on dist-test-slave-2cw1"
I20240502 14:24:30.920729 12379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:24:30.945403 12379 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:24:30.946017 12379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:24:30.946228 12379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:24:30.946969 12379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:24:30.948153 12379 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:24:30.948258 12379 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:30.948360 12379 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:24:30.948428 12379 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:30.976218 12379 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.69:33193
I20240502 14:24:30.976297 12506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.69:33193 every 8 connection(s)
I20240502 14:24:30.977257 12379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:24:30.986691 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 12379
I20240502 14:24:30.986883 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 14:24:30.987645 12507 heartbeater.cc:347] Connected to a master server at 127.5.107.126:36087
I20240502 14:24:30.987912 12507 heartbeater.cc:464] Registering TS with master...
I20240502 14:24:30.988629 12507 heartbeater.cc:511] Master 127.5.107.126:36087 requested a full tablet report, sending...
I20240502 14:24:30.989727 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.70:0
--local_ip_for_outbound_sockets=127.5.107.70
--webserver_interface=127.5.107.70
--webserver_port=0
--tserver_master_addrs=127.5.107.126:36087
--builtin_ntp_servers=127.5.107.84:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:24:30.989773 11793 ts_manager.cc:194] Registered new tserver with Master: 723fb53f34c94c6aa5f7ee3a65711268 (127.5.107.69:33193)
I20240502 14:24:30.990432 11793 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.69:45865
I20240502 14:24:31.198590 12245 heartbeater.cc:503] Master 127.5.107.126:36087 was elected leader, sending a full tablet report...
W20240502 14:24:31.278757 12510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:24:31.278972 12510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:24:31.279045 12510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:24:31.285499 12510 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:24:31.285584 12510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:24:31.285686 12510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.70
I20240502 14:24:31.293145 12510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.5.107.70
--webserver_port=0
--tserver_master_addrs=127.5.107.126:36087
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.70
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:24:31.294106 12510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:24:31.295475 12510 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:24:31.305459 12516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:31.306094 12515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:31.306631 12518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:24:31.307281 12510 server_base.cc:1034] running on GCE node
I20240502 14:24:31.307785 12510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:24:31.308557 12510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:24:31.309733 12510 hybrid_clock.cc:648] HybridClock initialized: now 1714659871309698 us; error 30 us; skew 500 ppm
I20240502 14:24:31.309986 12510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:24:31.311846 12510 webserver.cc:460] Webserver started at http://127.5.107.70:39797/ using document root <none> and password file <none>
I20240502 14:24:31.312500 12510 fs_manager.cc:359] Metadata directory not provided
I20240502 14:24:31.312604 12510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:24:31.312889 12510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:24:31.314989 12510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "375c6d5bdf924866b69b0e6667732692"
format_stamp: "Formatted at 2024-05-02 14:24:31 on dist-test-slave-2cw1"
I20240502 14:24:31.315605 12510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "375c6d5bdf924866b69b0e6667732692"
format_stamp: "Formatted at 2024-05-02 14:24:31 on dist-test-slave-2cw1"
I20240502 14:24:31.319782 12510 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:24:31.322783 12524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:31.324024 12510 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:24:31.324146 12510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "375c6d5bdf924866b69b0e6667732692"
format_stamp: "Formatted at 2024-05-02 14:24:31 on dist-test-slave-2cw1"
I20240502 14:24:31.324298 12510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:24:31.349946 12510 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:24:31.350657 12510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:24:31.350878 12510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:24:31.351663 12510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:24:31.353070 12510 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:24:31.353175 12510 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:31.353307 12510 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:24:31.353382 12510 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:31.381165 12510 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.70:33951
I20240502 14:24:31.381265 12637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.70:33951 every 8 connection(s)
I20240502 14:24:31.382346 12510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 14:24:31.384167 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 12510
I20240502 14:24:31.384320 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 14:24:31.387533 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.71:0
--local_ip_for_outbound_sockets=127.5.107.71
--webserver_interface=127.5.107.71
--webserver_port=0
--tserver_master_addrs=127.5.107.126:36087
--builtin_ntp_servers=127.5.107.84:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:24:31.395430 12638 heartbeater.cc:347] Connected to a master server at 127.5.107.126:36087
I20240502 14:24:31.395697 12638 heartbeater.cc:464] Registering TS with master...
I20240502 14:24:31.396301 12638 heartbeater.cc:511] Master 127.5.107.126:36087 requested a full tablet report, sending...
I20240502 14:24:31.397493 11793 ts_manager.cc:194] Registered new tserver with Master: 375c6d5bdf924866b69b0e6667732692 (127.5.107.70:33951)
I20240502 14:24:31.398164 11793 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.70:44579
I20240502 14:24:31.601985 12376 heartbeater.cc:503] Master 127.5.107.126:36087 was elected leader, sending a full tablet report...
W20240502 14:24:31.691186 12641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:24:31.691421 12641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:24:31.691496 12641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:24:31.698251 12641 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:24:31.698351 12641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:24:31.698460 12641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.71
I20240502 14:24:31.706370 12641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.5.107.71
--webserver_port=0
--tserver_master_addrs=127.5.107.126:36087
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.71
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:24:31.707414 12641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:24:31.709059 12641 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:24:31.720191 12647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:31.720307 12646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:31.720897 12649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:24:31.721515 12641 server_base.cc:1034] running on GCE node
I20240502 14:24:31.722013 12641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:24:31.722800 12641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:24:31.723978 12641 hybrid_clock.cc:648] HybridClock initialized: now 1714659871723931 us; error 44 us; skew 500 ppm
I20240502 14:24:31.724228 12641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:24:31.726215 12641 webserver.cc:460] Webserver started at http://127.5.107.71:38813/ using document root <none> and password file <none>
I20240502 14:24:31.726917 12641 fs_manager.cc:359] Metadata directory not provided
I20240502 14:24:31.727015 12641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:24:31.727313 12641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:24:31.729790 12641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "b0d1c78721a746e99d0d3056f1e31371"
format_stamp: "Formatted at 2024-05-02 14:24:31 on dist-test-slave-2cw1"
I20240502 14:24:31.730473 12641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "b0d1c78721a746e99d0d3056f1e31371"
format_stamp: "Formatted at 2024-05-02 14:24:31 on dist-test-slave-2cw1"
I20240502 14:24:31.734890 12641 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:24:31.737747 12655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:31.738888 12641 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:24:31.739020 12641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "b0d1c78721a746e99d0d3056f1e31371"
format_stamp: "Formatted at 2024-05-02 14:24:31 on dist-test-slave-2cw1"
I20240502 14:24:31.739178 12641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:24:31.753365 12641 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:24:31.754016 12641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:24:31.754235 12641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:24:31.755007 12641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:24:31.756201 12641 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:24:31.756289 12641 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:31.756387 12641 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:24:31.756451 12641 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:31.783547 12641 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.71:33799
I20240502 14:24:31.783599 12767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.71:33799 every 8 connection(s)
I20240502 14:24:31.784616 12641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 14:24:31.791076 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 12641
I20240502 14:24:31.791261 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 14:24:31.796463 12768 heartbeater.cc:347] Connected to a master server at 127.5.107.126:36087
I20240502 14:24:31.796687 12768 heartbeater.cc:464] Registering TS with master...
I20240502 14:24:31.797286 12768 heartbeater.cc:511] Master 127.5.107.126:36087 requested a full tablet report, sending...
I20240502 14:24:31.798393 11793 ts_manager.cc:194] Registered new tserver with Master: b0d1c78721a746e99d0d3056f1e31371 (127.5.107.71:33799)
I20240502 14:24:31.799058 11793 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.71:42999
I20240502 14:24:31.807109 5549 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 14:24:31.821326 11793 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60480:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 14:24:31.850447 12572 tablet_service.cc:1467] Processing CreateTablet for tablet f279550142c648d89b81feee9740f43e (DEFAULT_TABLE table=TestTable [id=06ce4d13691e489389354201980c0142]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:31.850448 11916 tablet_service.cc:1467] Processing CreateTablet for tablet f279550142c648d89b81feee9740f43e (DEFAULT_TABLE table=TestTable [id=06ce4d13691e489389354201980c0142]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:31.851518 11916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f279550142c648d89b81feee9740f43e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:31.851541 12572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f279550142c648d89b81feee9740f43e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:31.860780 12785 tablet_bootstrap.cc:492] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092: Bootstrap starting.
I20240502 14:24:31.860783 12784 tablet_bootstrap.cc:492] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692: Bootstrap starting.
I20240502 14:24:31.863757 12784 tablet_bootstrap.cc:654] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:31.863759 12785 tablet_bootstrap.cc:654] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:31.864570 12785 log.cc:824] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092: Log is configured to *not* fsync() on all Append() calls
I20240502 14:24:31.864570 12784 log.cc:824] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692: Log is configured to *not* fsync() on all Append() calls
I20240502 14:24:31.866737 12784 tablet_bootstrap.cc:492] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692: No bootstrap required, opened a new log
I20240502 14:24:31.866746 12785 tablet_bootstrap.cc:492] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092: No bootstrap required, opened a new log
I20240502 14:24:31.866983 12784 ts_tablet_manager.cc:1367] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 14:24:31.866986 12785 ts_tablet_manager.cc:1367] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 14:24:31.872182 12784 raft_consensus.cc:348] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:24:31.872180 12785 raft_consensus.cc:348] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:24:31.872462 12785 raft_consensus.cc:374] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:31.872473 12784 raft_consensus.cc:374] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:31.872563 12784 raft_consensus.cc:729] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 375c6d5bdf924866b69b0e6667732692, State: Initialized, Role: FOLLOWER
I20240502 14:24:31.872561 12785 raft_consensus.cc:729] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e41abe35d8f40a2a01953eea1c5d092, State: Initialized, Role: FOLLOWER
I20240502 14:24:31.873092 12785 consensus_queue.cc:260] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:24:31.873163 12784 consensus_queue.cc:260] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:24:31.874200 12638 heartbeater.cc:503] Master 127.5.107.126:36087 was elected leader, sending a full tablet report...
I20240502 14:24:31.874651 12785 ts_tablet_manager.cc:1398] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240502 14:24:31.874698 12784 ts_tablet_manager.cc:1398] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
W20240502 14:24:31.883911 12639 tablet.cc:2369] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:24:31.912947 11982 tablet.cc:2369] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:24:31.992422 12507 heartbeater.cc:503] Master 127.5.107.126:36087 was elected leader, sending a full tablet report...
I20240502 14:24:32.003571 12789 raft_consensus.cc:482] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:24:32.003787 12789 raft_consensus.cc:504] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:24:32.005097 12789 leader_election.cc:288] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871)
I20240502 14:24:32.010778 11936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f279550142c648d89b81feee9740f43e" candidate_uuid: "375c6d5bdf924866b69b0e6667732692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e41abe35d8f40a2a01953eea1c5d092" is_pre_election: true
I20240502 14:24:32.011196 11936 raft_consensus.cc:2454] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 375c6d5bdf924866b69b0e6667732692 in term 0.
I20240502 14:24:32.011713 12526 leader_election.cc:302] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e41abe35d8f40a2a01953eea1c5d092, 375c6d5bdf924866b69b0e6667732692; no voters:
I20240502 14:24:32.012079 12789 raft_consensus.cc:2781] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:24:32.012198 12789 raft_consensus.cc:482] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:24:32.012297 12789 raft_consensus.cc:3037] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:32.014256 12789 raft_consensus.cc:504] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:24:32.014665 12789 leader_election.cc:288] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [CANDIDATE]: Term 1 election: Requested vote from peers 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871)
I20240502 14:24:32.015036 11936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f279550142c648d89b81feee9740f43e" candidate_uuid: "375c6d5bdf924866b69b0e6667732692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e41abe35d8f40a2a01953eea1c5d092"
I20240502 14:24:32.015208 11936 raft_consensus.cc:3037] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:32.017026 11936 raft_consensus.cc:2454] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 375c6d5bdf924866b69b0e6667732692 in term 1.
I20240502 14:24:32.017402 12526 leader_election.cc:302] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e41abe35d8f40a2a01953eea1c5d092, 375c6d5bdf924866b69b0e6667732692; no voters:
I20240502 14:24:32.017640 12789 raft_consensus.cc:2781] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:24:32.018038 12789 raft_consensus.cc:686] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [term 1 LEADER]: Becoming Leader. State: Replica: 375c6d5bdf924866b69b0e6667732692, State: Running, Role: LEADER
I20240502 14:24:32.018460 12789 consensus_queue.cc:237] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:24:32.022053 11793 catalog_manager.cc:5484] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 reported cstate change: term changed from 0 to 1, leader changed from <none> to 375c6d5bdf924866b69b0e6667732692 (127.5.107.70). New cstate: current_term: 1 leader_uuid: "375c6d5bdf924866b69b0e6667732692" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } } }
I20240502 14:24:32.074604 5549 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 14:24:32.076393 5549 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0e41abe35d8f40a2a01953eea1c5d092 to finish bootstrapping
I20240502 14:24:32.084103 5549 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 36e66c7b1a03458b89a91c56a57df72a to finish bootstrapping
I20240502 14:24:32.091966 5549 ts_itest-base.cc:233] Waiting for 0 tablets on tserver d9748c575342401bb80683c7e2c4e105 to finish bootstrapping
I20240502 14:24:32.099191 5549 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 93c82e87d8af478292ffe1a3d3986049 to finish bootstrapping
I20240502 14:24:32.106719 5549 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 723fb53f34c94c6aa5f7ee3a65711268 to finish bootstrapping
I20240502 14:24:32.113658 5549 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 375c6d5bdf924866b69b0e6667732692 to finish bootstrapping
I20240502 14:24:32.119532 5549 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b0d1c78721a746e99d0d3056f1e31371 to finish bootstrapping
I20240502 14:24:32.127699 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 12248
I20240502 14:24:32.128687 12371 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:24:32.202745 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 12248
I20240502 14:24:32.218050 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 12379
I20240502 14:24:32.219084 12502 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:24:32.294428 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 12379
I20240502 14:24:32.308903 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 12510
I20240502 14:24:32.310075 12633 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:24:32.389753 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 12510
I20240502 14:24:32.405618 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 12641
I20240502 14:24:32.406651 12763 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:24:32.487375 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 12641
I20240502 14:24:34.255663 12804 raft_consensus.cc:482] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:24:34.255836 12804 raft_consensus.cc:504] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:24:34.257063 12804 leader_election.cc:288] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 375c6d5bdf924866b69b0e6667732692 (127.5.107.70:33951)
W20240502 14:24:34.257997 11871 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:33951: connect: Connection refused (error 111)
W20240502 14:24:34.259368 11871 leader_election.cc:334] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 375c6d5bdf924866b69b0e6667732692 (127.5.107.70:33951): Network error: Client connection negotiation failed: client connection to 127.5.107.70:33951: connect: Connection refused (error 111)
I20240502 14:24:34.259516 11871 leader_election.cc:302] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0e41abe35d8f40a2a01953eea1c5d092; no voters: 375c6d5bdf924866b69b0e6667732692
I20240502 14:24:34.259835 12804 raft_consensus.cc:2726] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:24:35.767778 12807 raft_consensus.cc:482] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:24:35.767956 12807 raft_consensus.cc:504] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:24:35.768481 12807 leader_election.cc:288] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 375c6d5bdf924866b69b0e6667732692 (127.5.107.70:33951)
W20240502 14:24:35.770642 11871 leader_election.cc:334] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 375c6d5bdf924866b69b0e6667732692 (127.5.107.70:33951): Network error: Client connection negotiation failed: client connection to 127.5.107.70:33951: connect: Connection refused (error 111)
I20240502 14:24:35.770754 11871 leader_election.cc:302] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0e41abe35d8f40a2a01953eea1c5d092; no voters: 375c6d5bdf924866b69b0e6667732692
I20240502 14:24:35.770959 12807 raft_consensus.cc:2726] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:24:36.253527 11793 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60480:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:24:36.264647 11916 tablet_service.cc:1467] Processing CreateTablet for tablet 4987edf6ce4d44d18162ff06e3f74e23 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1ebc3641cd6e480c85c60dd430ccb365]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:36.265269 11916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4987edf6ce4d44d18162ff06e3f74e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:36.277069 12818 tablet_bootstrap.cc:492] T 4987edf6ce4d44d18162ff06e3f74e23 P 0e41abe35d8f40a2a01953eea1c5d092: Bootstrap starting.
I20240502 14:24:36.278677 12818 tablet_bootstrap.cc:654] T 4987edf6ce4d44d18162ff06e3f74e23 P 0e41abe35d8f40a2a01953eea1c5d092: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:36.282608 12818 tablet_bootstrap.cc:492] T 4987edf6ce4d44d18162ff06e3f74e23 P 0e41abe35d8f40a2a01953eea1c5d092: No bootstrap required, opened a new log
I20240502 14:24:36.283003 12818 ts_tablet_manager.cc:1367] T 4987edf6ce4d44d18162ff06e3f74e23 P 0e41abe35d8f40a2a01953eea1c5d092: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 14:24:36.285938 12045 tablet_service.cc:1467] Processing CreateTablet for tablet 4987edf6ce4d44d18162ff06e3f74e23 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1ebc3641cd6e480c85c60dd430ccb365]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:36.286279 12179 tablet_service.cc:1467] Processing CreateTablet for tablet c3811b0d595f4292b9d96454deba8156 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1ebc3641cd6e480c85c60dd430ccb365]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:36.287015 12045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4987edf6ce4d44d18162ff06e3f74e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:36.287279 12179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3811b0d595f4292b9d96454deba8156. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:36.293378 12047 tablet_service.cc:1467] Processing CreateTablet for tablet c3811b0d595f4292b9d96454deba8156 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1ebc3641cd6e480c85c60dd430ccb365]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:36.293506 12046 tablet_service.cc:1467] Processing CreateTablet for tablet e05a3a4fb76d4c46908945ad0d838509 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1ebc3641cd6e480c85c60dd430ccb365]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:36.293651 12178 tablet_service.cc:1467] Processing CreateTablet for tablet e05a3a4fb76d4c46908945ad0d838509 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1ebc3641cd6e480c85c60dd430ccb365]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:36.293835 12047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3811b0d595f4292b9d96454deba8156. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:36.294044 12178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e05a3a4fb76d4c46908945ad0d838509. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:36.294064 12046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e05a3a4fb76d4c46908945ad0d838509. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:36.294306 12818 raft_consensus.cc:348] T 4987edf6ce4d44d18162ff06e3f74e23 P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:36.294483 12818 raft_consensus.cc:374] T 4987edf6ce4d44d18162ff06e3f74e23 P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:36.294559 12818 raft_consensus.cc:729] T 4987edf6ce4d44d18162ff06e3f74e23 P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e41abe35d8f40a2a01953eea1c5d092, State: Initialized, Role: FOLLOWER
I20240502 14:24:36.294750 12818 consensus_queue.cc:260] T 4987edf6ce4d44d18162ff06e3f74e23 P 0e41abe35d8f40a2a01953eea1c5d092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:36.302888 12821 tablet_bootstrap.cc:492] T e05a3a4fb76d4c46908945ad0d838509 P d9748c575342401bb80683c7e2c4e105: Bootstrap starting.
I20240502 14:24:36.305892 12818 ts_tablet_manager.cc:1398] T 4987edf6ce4d44d18162ff06e3f74e23 P 0e41abe35d8f40a2a01953eea1c5d092: Time spent starting tablet: real 0.023s user 0.003s sys 0.009s
I20240502 14:24:36.306744 12822 tablet_bootstrap.cc:492] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a: Bootstrap starting.
I20240502 14:24:36.308368 12821 tablet_bootstrap.cc:654] T e05a3a4fb76d4c46908945ad0d838509 P d9748c575342401bb80683c7e2c4e105: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:36.309367 12821 log.cc:824] T e05a3a4fb76d4c46908945ad0d838509 P d9748c575342401bb80683c7e2c4e105: Log is configured to *not* fsync() on all Append() calls
I20240502 14:24:36.309391 12822 tablet_bootstrap.cc:654] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:36.310482 12822 log.cc:824] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a: Log is configured to *not* fsync() on all Append() calls
I20240502 14:24:36.311686 12821 tablet_bootstrap.cc:492] T e05a3a4fb76d4c46908945ad0d838509 P d9748c575342401bb80683c7e2c4e105: No bootstrap required, opened a new log
I20240502 14:24:36.311897 12821 ts_tablet_manager.cc:1367] T e05a3a4fb76d4c46908945ad0d838509 P d9748c575342401bb80683c7e2c4e105: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240502 14:24:36.312532 12822 tablet_bootstrap.cc:492] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a: No bootstrap required, opened a new log
I20240502 14:24:36.312734 12822 ts_tablet_manager.cc:1367] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20240502 14:24:36.317561 12821 raft_consensus.cc:348] T e05a3a4fb76d4c46908945ad0d838509 P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:36.317868 12821 raft_consensus.cc:374] T e05a3a4fb76d4c46908945ad0d838509 P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:36.317955 12821 raft_consensus.cc:729] T e05a3a4fb76d4c46908945ad0d838509 P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9748c575342401bb80683c7e2c4e105, State: Initialized, Role: FOLLOWER
I20240502 14:24:36.318279 12822 raft_consensus.cc:348] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:36.318552 12822 raft_consensus.cc:374] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:36.318645 12822 raft_consensus.cc:729] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36e66c7b1a03458b89a91c56a57df72a, State: Initialized, Role: FOLLOWER
I20240502 14:24:36.318601 12821 consensus_queue.cc:260] T e05a3a4fb76d4c46908945ad0d838509 P d9748c575342401bb80683c7e2c4e105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:36.319242 12822 consensus_queue.cc:260] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:36.320092 12821 ts_tablet_manager.cc:1398] T e05a3a4fb76d4c46908945ad0d838509 P d9748c575342401bb80683c7e2c4e105: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20240502 14:24:36.320462 12821 tablet_bootstrap.cc:492] T c3811b0d595f4292b9d96454deba8156 P d9748c575342401bb80683c7e2c4e105: Bootstrap starting.
I20240502 14:24:36.321846 12821 tablet_bootstrap.cc:654] T c3811b0d595f4292b9d96454deba8156 P d9748c575342401bb80683c7e2c4e105: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:36.321856 12822 ts_tablet_manager.cc:1398] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20240502 14:24:36.322322 12822 tablet_bootstrap.cc:492] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a: Bootstrap starting.
I20240502 14:24:36.323559 12821 tablet_bootstrap.cc:492] T c3811b0d595f4292b9d96454deba8156 P d9748c575342401bb80683c7e2c4e105: No bootstrap required, opened a new log
I20240502 14:24:36.323765 12821 ts_tablet_manager.cc:1367] T c3811b0d595f4292b9d96454deba8156 P d9748c575342401bb80683c7e2c4e105: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240502 14:24:36.323977 12822 tablet_bootstrap.cc:654] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:36.324656 12821 raft_consensus.cc:348] T c3811b0d595f4292b9d96454deba8156 P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:36.324796 12821 raft_consensus.cc:374] T c3811b0d595f4292b9d96454deba8156 P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:36.324919 12821 raft_consensus.cc:729] T c3811b0d595f4292b9d96454deba8156 P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9748c575342401bb80683c7e2c4e105, State: Initialized, Role: FOLLOWER
I20240502 14:24:36.325115 12821 consensus_queue.cc:260] T c3811b0d595f4292b9d96454deba8156 P d9748c575342401bb80683c7e2c4e105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:36.325699 12821 ts_tablet_manager.cc:1398] T c3811b0d595f4292b9d96454deba8156 P d9748c575342401bb80683c7e2c4e105: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:24:36.327023 12822 tablet_bootstrap.cc:492] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a: No bootstrap required, opened a new log
I20240502 14:24:36.327167 12822 ts_tablet_manager.cc:1367] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 14:24:36.327893 12822 raft_consensus.cc:348] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:36.328022 12822 raft_consensus.cc:374] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:36.328088 12822 raft_consensus.cc:729] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36e66c7b1a03458b89a91c56a57df72a, State: Initialized, Role: FOLLOWER
I20240502 14:24:36.328339 12822 consensus_queue.cc:260] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:36.328783 12822 ts_tablet_manager.cc:1398] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:24:36.329097 12822 tablet_bootstrap.cc:492] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a: Bootstrap starting.
I20240502 14:24:36.330789 12822 tablet_bootstrap.cc:654] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:36.332430 12822 tablet_bootstrap.cc:492] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a: No bootstrap required, opened a new log
I20240502 14:24:36.332564 12822 ts_tablet_manager.cc:1367] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:24:36.333061 12822 raft_consensus.cc:348] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:36.333266 12822 raft_consensus.cc:374] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:36.333333 12822 raft_consensus.cc:729] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36e66c7b1a03458b89a91c56a57df72a, State: Initialized, Role: FOLLOWER
I20240502 14:24:36.333539 12822 consensus_queue.cc:260] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:36.333943 12822 ts_tablet_manager.cc:1398] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:24:36.345919 12826 raft_consensus.cc:482] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:24:36.346134 12826 raft_consensus.cc:504] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:36.347636 12826 leader_election.cc:288] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871)
I20240502 14:24:36.353257 11936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4987edf6ce4d44d18162ff06e3f74e23" candidate_uuid: "36e66c7b1a03458b89a91c56a57df72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e41abe35d8f40a2a01953eea1c5d092" is_pre_election: true
I20240502 14:24:36.353464 11936 raft_consensus.cc:2454] T 4987edf6ce4d44d18162ff06e3f74e23 P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36e66c7b1a03458b89a91c56a57df72a in term 0.
I20240502 14:24:36.353967 12000 leader_election.cc:302] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e41abe35d8f40a2a01953eea1c5d092, 36e66c7b1a03458b89a91c56a57df72a; no voters:
I20240502 14:24:36.354326 12826 raft_consensus.cc:2781] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:24:36.354457 12826 raft_consensus.cc:482] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:24:36.354552 12826 raft_consensus.cc:3037] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:36.356400 12826 raft_consensus.cc:504] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:36.356806 12826 leader_election.cc:288] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 election: Requested vote from peers 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871)
I20240502 14:24:36.357160 11936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4987edf6ce4d44d18162ff06e3f74e23" candidate_uuid: "36e66c7b1a03458b89a91c56a57df72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e41abe35d8f40a2a01953eea1c5d092"
I20240502 14:24:36.357362 11936 raft_consensus.cc:3037] T 4987edf6ce4d44d18162ff06e3f74e23 P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:36.359098 11936 raft_consensus.cc:2454] T 4987edf6ce4d44d18162ff06e3f74e23 P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36e66c7b1a03458b89a91c56a57df72a in term 1.
I20240502 14:24:36.359395 12000 leader_election.cc:302] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e41abe35d8f40a2a01953eea1c5d092, 36e66c7b1a03458b89a91c56a57df72a; no voters:
I20240502 14:24:36.359622 12826 raft_consensus.cc:2781] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:24:36.360013 12826 raft_consensus.cc:686] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Becoming Leader. State: Replica: 36e66c7b1a03458b89a91c56a57df72a, State: Running, Role: LEADER
I20240502 14:24:36.360436 12826 consensus_queue.cc:237] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:36.364046 11792 catalog_manager.cc:5484] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a reported cstate change: term changed from 0 to 1, leader changed from <none> to 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66). New cstate: current_term: 1 leader_uuid: "36e66c7b1a03458b89a91c56a57df72a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } }
I20240502 14:24:36.414458 12826 raft_consensus.cc:482] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:24:36.414655 12826 raft_consensus.cc:504] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:36.415251 12826 leader_election.cc:288] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9748c575342401bb80683c7e2c4e105 (127.5.107.67:45923)
I20240502 14:24:36.420456 12200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05a3a4fb76d4c46908945ad0d838509" candidate_uuid: "36e66c7b1a03458b89a91c56a57df72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9748c575342401bb80683c7e2c4e105" is_pre_election: true
I20240502 14:24:36.420956 12200 raft_consensus.cc:2454] T e05a3a4fb76d4c46908945ad0d838509 P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36e66c7b1a03458b89a91c56a57df72a in term 0.
I20240502 14:24:36.421465 11999 leader_election.cc:302] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 36e66c7b1a03458b89a91c56a57df72a, d9748c575342401bb80683c7e2c4e105; no voters:
I20240502 14:24:36.421730 12826 raft_consensus.cc:2781] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:24:36.421821 12826 raft_consensus.cc:482] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:24:36.421907 12826 raft_consensus.cc:3037] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:36.423730 12826 raft_consensus.cc:504] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:36.424134 12826 leader_election.cc:288] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 election: Requested vote from peers d9748c575342401bb80683c7e2c4e105 (127.5.107.67:45923)
I20240502 14:24:36.424530 12200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05a3a4fb76d4c46908945ad0d838509" candidate_uuid: "36e66c7b1a03458b89a91c56a57df72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9748c575342401bb80683c7e2c4e105"
I20240502 14:24:36.424667 12200 raft_consensus.cc:3037] T e05a3a4fb76d4c46908945ad0d838509 P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:36.426518 12200 raft_consensus.cc:2454] T e05a3a4fb76d4c46908945ad0d838509 P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36e66c7b1a03458b89a91c56a57df72a in term 1.
I20240502 14:24:36.426821 11999 leader_election.cc:302] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 36e66c7b1a03458b89a91c56a57df72a, d9748c575342401bb80683c7e2c4e105; no voters:
I20240502 14:24:36.427057 12826 raft_consensus.cc:2781] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:24:36.427191 12826 raft_consensus.cc:686] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Becoming Leader. State: Replica: 36e66c7b1a03458b89a91c56a57df72a, State: Running, Role: LEADER
I20240502 14:24:36.427407 12826 consensus_queue.cc:237] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:36.429368 11792 catalog_manager.cc:5484] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a reported cstate change: term changed from 0 to 1, leader changed from <none> to 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66). New cstate: current_term: 1 leader_uuid: "36e66c7b1a03458b89a91c56a57df72a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } }
I20240502 14:24:36.432960 12826 raft_consensus.cc:482] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:24:36.433060 12826 raft_consensus.cc:504] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:36.433430 12826 leader_election.cc:288] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9748c575342401bb80683c7e2c4e105 (127.5.107.67:45923)
I20240502 14:24:36.433797 12200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3811b0d595f4292b9d96454deba8156" candidate_uuid: "36e66c7b1a03458b89a91c56a57df72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9748c575342401bb80683c7e2c4e105" is_pre_election: true
I20240502 14:24:36.433962 12200 raft_consensus.cc:2454] T c3811b0d595f4292b9d96454deba8156 P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36e66c7b1a03458b89a91c56a57df72a in term 0.
I20240502 14:24:36.434242 11999 leader_election.cc:302] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 36e66c7b1a03458b89a91c56a57df72a, d9748c575342401bb80683c7e2c4e105; no voters:
I20240502 14:24:36.434450 12826 raft_consensus.cc:2781] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:24:36.434532 12826 raft_consensus.cc:482] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:24:36.434609 12826 raft_consensus.cc:3037] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:36.436055 12826 raft_consensus.cc:504] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:36.436370 12826 leader_election.cc:288] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 election: Requested vote from peers d9748c575342401bb80683c7e2c4e105 (127.5.107.67:45923)
I20240502 14:24:36.436731 12200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3811b0d595f4292b9d96454deba8156" candidate_uuid: "36e66c7b1a03458b89a91c56a57df72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9748c575342401bb80683c7e2c4e105"
I20240502 14:24:36.436913 12200 raft_consensus.cc:3037] T c3811b0d595f4292b9d96454deba8156 P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:36.438483 12200 raft_consensus.cc:2454] T c3811b0d595f4292b9d96454deba8156 P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36e66c7b1a03458b89a91c56a57df72a in term 1.
W20240502 14:24:36.438750 12246 tablet.cc:2369] T c3811b0d595f4292b9d96454deba8156 P d9748c575342401bb80683c7e2c4e105: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:24:36.438789 11999 leader_election.cc:302] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 36e66c7b1a03458b89a91c56a57df72a, d9748c575342401bb80683c7e2c4e105; no voters:
I20240502 14:24:36.439064 12826 raft_consensus.cc:2781] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:24:36.439195 12826 raft_consensus.cc:686] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Becoming Leader. State: Replica: 36e66c7b1a03458b89a91c56a57df72a, State: Running, Role: LEADER
I20240502 14:24:36.439404 12826 consensus_queue.cc:237] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:36.441179 11792 catalog_manager.cc:5484] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a reported cstate change: term changed from 0 to 1, leader changed from <none> to 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66). New cstate: current_term: 1 leader_uuid: "36e66c7b1a03458b89a91c56a57df72a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } }
W20240502 14:24:36.561542 12114 tablet.cc:2369] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:24:36.780236 12826 consensus_queue.cc:1035] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:24:36.808333 12837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:24:36.808694 12837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:24:36.816135 12837 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 14:24:36.874466 12833 consensus_queue.cc:1035] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:24:36.966221 12826 consensus_queue.cc:1035] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 178.31 ms
time per row: 0.0891548 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:24:37.967775 11792 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60480:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:24:37.975589 11916 tablet_service.cc:1467] Processing CreateTablet for tablet 9069116816104c79819b25838836575b (DEFAULT_TABLE table=rebalance_test_table_1 [id=fe50532b3c3d46b7bdae21e985f105f6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:37.975697 12178 tablet_service.cc:1467] Processing CreateTablet for tablet 9069116816104c79819b25838836575b (DEFAULT_TABLE table=rebalance_test_table_1 [id=fe50532b3c3d46b7bdae21e985f105f6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:37.976081 11916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9069116816104c79819b25838836575b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:37.976073 12178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9069116816104c79819b25838836575b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:37.976574 11914 tablet_service.cc:1467] Processing CreateTablet for tablet 64e9d77a5b9343489db94f4988180213 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fe50532b3c3d46b7bdae21e985f105f6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:37.976955 11914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64e9d77a5b9343489db94f4988180213. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:37.977030 11915 tablet_service.cc:1467] Processing CreateTablet for tablet bb1b842f76d74f6e996e48c245deda9b (DEFAULT_TABLE table=rebalance_test_table_1 [id=fe50532b3c3d46b7bdae21e985f105f6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:37.977448 11915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb1b842f76d74f6e996e48c245deda9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:37.977528 12179 tablet_service.cc:1467] Processing CreateTablet for tablet bb1b842f76d74f6e996e48c245deda9b (DEFAULT_TABLE table=rebalance_test_table_1 [id=fe50532b3c3d46b7bdae21e985f105f6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:37.977864 12179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb1b842f76d74f6e996e48c245deda9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:37.982017 12046 tablet_service.cc:1467] Processing CreateTablet for tablet 64e9d77a5b9343489db94f4988180213 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fe50532b3c3d46b7bdae21e985f105f6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:37.982528 12046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64e9d77a5b9343489db94f4988180213. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:37.989513 12889 tablet_bootstrap.cc:492] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105: Bootstrap starting.
I20240502 14:24:37.990223 12891 tablet_bootstrap.cc:492] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a: Bootstrap starting.
I20240502 14:24:37.991391 12889 tablet_bootstrap.cc:654] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:37.991631 12891 tablet_bootstrap.cc:654] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:37.995558 12890 tablet_bootstrap.cc:492] T 64e9d77a5b9343489db94f4988180213 P 0e41abe35d8f40a2a01953eea1c5d092: Bootstrap starting.
I20240502 14:24:37.997033 12889 tablet_bootstrap.cc:492] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105: No bootstrap required, opened a new log
I20240502 14:24:37.997164 12889 ts_tablet_manager.cc:1367] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20240502 14:24:37.997231 12891 tablet_bootstrap.cc:492] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a: No bootstrap required, opened a new log
I20240502 14:24:37.997368 12891 ts_tablet_manager.cc:1367] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.005s
I20240502 14:24:37.997413 12890 tablet_bootstrap.cc:654] T 64e9d77a5b9343489db94f4988180213 P 0e41abe35d8f40a2a01953eea1c5d092: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:37.997767 12889 raft_consensus.cc:348] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:37.997900 12889 raft_consensus.cc:374] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:37.997977 12889 raft_consensus.cc:729] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9748c575342401bb80683c7e2c4e105, State: Initialized, Role: FOLLOWER
I20240502 14:24:37.998076 12891 raft_consensus.cc:348] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:37.998145 12889 consensus_queue.cc:260] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:37.998255 12891 raft_consensus.cc:374] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:37.998330 12891 raft_consensus.cc:729] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36e66c7b1a03458b89a91c56a57df72a, State: Initialized, Role: FOLLOWER
I20240502 14:24:37.998548 12891 consensus_queue.cc:260] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:37.998842 12889 ts_tablet_manager.cc:1398] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:24:37.999051 12891 ts_tablet_manager.cc:1398] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:24:37.999105 12889 tablet_bootstrap.cc:492] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105: Bootstrap starting.
I20240502 14:24:37.999140 12890 tablet_bootstrap.cc:492] T 64e9d77a5b9343489db94f4988180213 P 0e41abe35d8f40a2a01953eea1c5d092: No bootstrap required, opened a new log
I20240502 14:24:37.999245 12890 ts_tablet_manager.cc:1367] T 64e9d77a5b9343489db94f4988180213 P 0e41abe35d8f40a2a01953eea1c5d092: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:24:37.999869 12890 raft_consensus.cc:348] T 64e9d77a5b9343489db94f4988180213 P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:38.000044 12890 raft_consensus.cc:374] T 64e9d77a5b9343489db94f4988180213 P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:38.000141 12890 raft_consensus.cc:729] T 64e9d77a5b9343489db94f4988180213 P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e41abe35d8f40a2a01953eea1c5d092, State: Initialized, Role: FOLLOWER
I20240502 14:24:38.000411 12889 tablet_bootstrap.cc:654] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:38.000332 12890 consensus_queue.cc:260] T 64e9d77a5b9343489db94f4988180213 P 0e41abe35d8f40a2a01953eea1c5d092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:38.001598 12890 ts_tablet_manager.cc:1398] T 64e9d77a5b9343489db94f4988180213 P 0e41abe35d8f40a2a01953eea1c5d092: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:24:38.001878 12890 tablet_bootstrap.cc:492] T bb1b842f76d74f6e996e48c245deda9b P 0e41abe35d8f40a2a01953eea1c5d092: Bootstrap starting.
I20240502 14:24:38.004081 12890 tablet_bootstrap.cc:654] T bb1b842f76d74f6e996e48c245deda9b P 0e41abe35d8f40a2a01953eea1c5d092: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:38.005946 12890 tablet_bootstrap.cc:492] T bb1b842f76d74f6e996e48c245deda9b P 0e41abe35d8f40a2a01953eea1c5d092: No bootstrap required, opened a new log
I20240502 14:24:38.006116 12890 ts_tablet_manager.cc:1367] T bb1b842f76d74f6e996e48c245deda9b P 0e41abe35d8f40a2a01953eea1c5d092: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:24:38.006266 12889 tablet_bootstrap.cc:492] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105: No bootstrap required, opened a new log
I20240502 14:24:38.006422 12889 ts_tablet_manager.cc:1367] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240502 14:24:38.006754 12890 raft_consensus.cc:348] T bb1b842f76d74f6e996e48c245deda9b P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } }
I20240502 14:24:38.006904 12890 raft_consensus.cc:374] T bb1b842f76d74f6e996e48c245deda9b P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:38.006999 12890 raft_consensus.cc:729] T bb1b842f76d74f6e996e48c245deda9b P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e41abe35d8f40a2a01953eea1c5d092, State: Initialized, Role: FOLLOWER
I20240502 14:24:38.007232 12890 consensus_queue.cc:260] T bb1b842f76d74f6e996e48c245deda9b P 0e41abe35d8f40a2a01953eea1c5d092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } }
I20240502 14:24:38.007220 12889 raft_consensus.cc:348] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } }
I20240502 14:24:38.007398 12889 raft_consensus.cc:374] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:38.007483 12889 raft_consensus.cc:729] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9748c575342401bb80683c7e2c4e105, State: Initialized, Role: FOLLOWER
I20240502 14:24:38.007675 12890 ts_tablet_manager.cc:1398] T bb1b842f76d74f6e996e48c245deda9b P 0e41abe35d8f40a2a01953eea1c5d092: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:24:38.007701 12889 consensus_queue.cc:260] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } }
I20240502 14:24:38.007931 12890 tablet_bootstrap.cc:492] T 9069116816104c79819b25838836575b P 0e41abe35d8f40a2a01953eea1c5d092: Bootstrap starting.
I20240502 14:24:38.008137 12889 ts_tablet_manager.cc:1398] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:24:38.009670 12890 tablet_bootstrap.cc:654] T 9069116816104c79819b25838836575b P 0e41abe35d8f40a2a01953eea1c5d092: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:38.011651 12890 tablet_bootstrap.cc:492] T 9069116816104c79819b25838836575b P 0e41abe35d8f40a2a01953eea1c5d092: No bootstrap required, opened a new log
I20240502 14:24:38.011785 12890 ts_tablet_manager.cc:1367] T 9069116816104c79819b25838836575b P 0e41abe35d8f40a2a01953eea1c5d092: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:24:38.012485 12890 raft_consensus.cc:348] T 9069116816104c79819b25838836575b P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:38.012624 12890 raft_consensus.cc:374] T 9069116816104c79819b25838836575b P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:38.012718 12890 raft_consensus.cc:729] T 9069116816104c79819b25838836575b P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e41abe35d8f40a2a01953eea1c5d092, State: Initialized, Role: FOLLOWER
I20240502 14:24:38.012909 12890 consensus_queue.cc:260] T 9069116816104c79819b25838836575b P 0e41abe35d8f40a2a01953eea1c5d092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:38.013335 12890 ts_tablet_manager.cc:1398] T 9069116816104c79819b25838836575b P 0e41abe35d8f40a2a01953eea1c5d092: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:24:38.026444 12895 raft_consensus.cc:482] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:24:38.026588 12895 raft_consensus.cc:504] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:38.027814 12895 leader_election.cc:288] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871)
I20240502 14:24:38.033387 11936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9069116816104c79819b25838836575b" candidate_uuid: "d9748c575342401bb80683c7e2c4e105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e41abe35d8f40a2a01953eea1c5d092" is_pre_election: true
I20240502 14:24:38.033562 11936 raft_consensus.cc:2454] T 9069116816104c79819b25838836575b P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9748c575342401bb80683c7e2c4e105 in term 0.
I20240502 14:24:38.034005 12133 leader_election.cc:302] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e41abe35d8f40a2a01953eea1c5d092, d9748c575342401bb80683c7e2c4e105; no voters:
I20240502 14:24:38.034319 12895 raft_consensus.cc:2781] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:24:38.034427 12895 raft_consensus.cc:482] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:24:38.034518 12895 raft_consensus.cc:3037] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:38.036355 12895 raft_consensus.cc:504] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:38.036763 12895 leader_election.cc:288] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105 [CANDIDATE]: Term 1 election: Requested vote from peers 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871)
I20240502 14:24:38.037132 11936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9069116816104c79819b25838836575b" candidate_uuid: "d9748c575342401bb80683c7e2c4e105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e41abe35d8f40a2a01953eea1c5d092"
I20240502 14:24:38.037328 11936 raft_consensus.cc:3037] T 9069116816104c79819b25838836575b P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:38.039166 11936 raft_consensus.cc:2454] T 9069116816104c79819b25838836575b P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9748c575342401bb80683c7e2c4e105 in term 1.
I20240502 14:24:38.039499 12133 leader_election.cc:302] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e41abe35d8f40a2a01953eea1c5d092, d9748c575342401bb80683c7e2c4e105; no voters:
I20240502 14:24:38.039783 12895 raft_consensus.cc:2781] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:24:38.040169 12895 raft_consensus.cc:686] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105 [term 1 LEADER]: Becoming Leader. State: Replica: d9748c575342401bb80683c7e2c4e105, State: Running, Role: LEADER
I20240502 14:24:38.040819 12895 consensus_queue.cc:237] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:38.043648 11793 catalog_manager.cc:5484] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105 reported cstate change: term changed from 0 to 1, leader changed from <none> to d9748c575342401bb80683c7e2c4e105 (127.5.107.67). New cstate: current_term: 1 leader_uuid: "d9748c575342401bb80683c7e2c4e105" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } }
I20240502 14:24:38.045257 12833 raft_consensus.cc:482] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:24:38.045421 12833 raft_consensus.cc:504] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:38.045989 12833 leader_election.cc:288] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871)
I20240502 14:24:38.046573 11936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64e9d77a5b9343489db94f4988180213" candidate_uuid: "36e66c7b1a03458b89a91c56a57df72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e41abe35d8f40a2a01953eea1c5d092" is_pre_election: true
I20240502 14:24:38.046752 11936 raft_consensus.cc:2454] T 64e9d77a5b9343489db94f4988180213 P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36e66c7b1a03458b89a91c56a57df72a in term 0.
I20240502 14:24:38.047086 12000 leader_election.cc:302] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e41abe35d8f40a2a01953eea1c5d092, 36e66c7b1a03458b89a91c56a57df72a; no voters:
I20240502 14:24:38.047331 12833 raft_consensus.cc:2781] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:24:38.047420 12833 raft_consensus.cc:482] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:24:38.047518 12833 raft_consensus.cc:3037] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:38.049144 12833 raft_consensus.cc:504] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:38.049571 12833 leader_election.cc:288] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 election: Requested vote from peers 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871)
I20240502 14:24:38.049916 11936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64e9d77a5b9343489db94f4988180213" candidate_uuid: "36e66c7b1a03458b89a91c56a57df72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e41abe35d8f40a2a01953eea1c5d092"
I20240502 14:24:38.050081 11936 raft_consensus.cc:3037] T 64e9d77a5b9343489db94f4988180213 P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:38.051671 11936 raft_consensus.cc:2454] T 64e9d77a5b9343489db94f4988180213 P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36e66c7b1a03458b89a91c56a57df72a in term 1.
I20240502 14:24:38.051971 12000 leader_election.cc:302] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e41abe35d8f40a2a01953eea1c5d092, 36e66c7b1a03458b89a91c56a57df72a; no voters:
I20240502 14:24:38.052212 12833 raft_consensus.cc:2781] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:24:38.052541 12833 raft_consensus.cc:686] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Becoming Leader. State: Replica: 36e66c7b1a03458b89a91c56a57df72a, State: Running, Role: LEADER
I20240502 14:24:38.052803 12833 consensus_queue.cc:237] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:38.054807 11793 catalog_manager.cc:5484] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a reported cstate change: term changed from 0 to 1, leader changed from <none> to 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66). New cstate: current_term: 1 leader_uuid: "36e66c7b1a03458b89a91c56a57df72a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } }
I20240502 14:24:38.201356 12895 raft_consensus.cc:482] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:24:38.201575 12895 raft_consensus.cc:504] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } }
I20240502 14:24:38.202059 12895 leader_election.cc:288] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871)
I20240502 14:24:38.202584 11936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb1b842f76d74f6e996e48c245deda9b" candidate_uuid: "d9748c575342401bb80683c7e2c4e105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e41abe35d8f40a2a01953eea1c5d092" is_pre_election: true
I20240502 14:24:38.202809 11936 raft_consensus.cc:2454] T bb1b842f76d74f6e996e48c245deda9b P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9748c575342401bb80683c7e2c4e105 in term 0.
I20240502 14:24:38.203229 12133 leader_election.cc:302] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e41abe35d8f40a2a01953eea1c5d092, d9748c575342401bb80683c7e2c4e105; no voters:
I20240502 14:24:38.203491 12895 raft_consensus.cc:2781] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:24:38.203580 12895 raft_consensus.cc:482] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:24:38.203698 12895 raft_consensus.cc:3037] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:38.205339 12895 raft_consensus.cc:504] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } }
I20240502 14:24:38.205711 12895 leader_election.cc:288] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 [CANDIDATE]: Term 1 election: Requested vote from peers 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871)
I20240502 14:24:38.206111 11936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb1b842f76d74f6e996e48c245deda9b" candidate_uuid: "d9748c575342401bb80683c7e2c4e105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e41abe35d8f40a2a01953eea1c5d092"
I20240502 14:24:38.206245 11936 raft_consensus.cc:3037] T bb1b842f76d74f6e996e48c245deda9b P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:38.207763 11936 raft_consensus.cc:2454] T bb1b842f76d74f6e996e48c245deda9b P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9748c575342401bb80683c7e2c4e105 in term 1.
I20240502 14:24:38.208076 12133 leader_election.cc:302] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e41abe35d8f40a2a01953eea1c5d092, d9748c575342401bb80683c7e2c4e105; no voters:
I20240502 14:24:38.208268 12895 raft_consensus.cc:2781] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:24:38.208401 12895 raft_consensus.cc:686] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 [term 1 LEADER]: Becoming Leader. State: Replica: d9748c575342401bb80683c7e2c4e105, State: Running, Role: LEADER
I20240502 14:24:38.208639 12895 consensus_queue.cc:237] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } }
I20240502 14:24:38.210729 11793 catalog_manager.cc:5484] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 reported cstate change: term changed from 0 to 1, leader changed from <none> to d9748c575342401bb80683c7e2c4e105 (127.5.107.67). New cstate: current_term: 1 leader_uuid: "d9748c575342401bb80683c7e2c4e105" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } }
I20240502 14:24:38.437044 12895 consensus_queue.cc:1035] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:24:38.528424 12833 consensus_queue.cc:1035] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:24:38.570223 12908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:24:38.570483 12908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:24:38.577090 12908 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 14:24:38.695647 11936 raft_consensus.cc:1261] T bb1b842f76d74f6e996e48c245deda9b P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Refusing update from remote peer d9748c575342401bb80683c7e2c4e105: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:24:38.696506 12903 consensus_queue.cc:1035] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 147.64 ms
time per row: 0.0738201 ms
I20240502 14:24:38.807811 12896 raft_consensus.cc:482] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:24:38.808003 12896 raft_consensus.cc:504] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:24:38.808643 12896 leader_election.cc:288] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 375c6d5bdf924866b69b0e6667732692 (127.5.107.70:33951)
W20240502 14:24:38.810726 11871 leader_election.cc:334] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 375c6d5bdf924866b69b0e6667732692 (127.5.107.70:33951): Network error: Client connection negotiation failed: client connection to 127.5.107.70:33951: connect: Connection refused (error 111)
I20240502 14:24:38.810874 11871 leader_election.cc:302] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0e41abe35d8f40a2a01953eea1c5d092; no voters: 375c6d5bdf924866b69b0e6667732692
I20240502 14:24:38.811092 12896 raft_consensus.cc:2726] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:24:39.699326 11787 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60480:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:24:39.707645 12046 tablet_service.cc:1467] Processing CreateTablet for tablet 003642773a724534847246de230ea34b (DEFAULT_TABLE table=rebalance_test_table_2 [id=ec56bc8cfbc04f26b92384c442eac5b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:39.707746 12178 tablet_service.cc:1467] Processing CreateTablet for tablet 003642773a724534847246de230ea34b (DEFAULT_TABLE table=rebalance_test_table_2 [id=ec56bc8cfbc04f26b92384c442eac5b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:39.708125 12046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 003642773a724534847246de230ea34b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:39.708158 12178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 003642773a724534847246de230ea34b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:39.708511 12045 tablet_service.cc:1467] Processing CreateTablet for tablet e7732ff4fa2744db94d0f8cb888de8d8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ec56bc8cfbc04f26b92384c442eac5b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:39.708576 11914 tablet_service.cc:1467] Processing CreateTablet for tablet 439f0129f4bd49b7b582be22be83e0b4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ec56bc8cfbc04f26b92384c442eac5b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:39.708845 12045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7732ff4fa2744db94d0f8cb888de8d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:39.708968 11914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 439f0129f4bd49b7b582be22be83e0b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:39.710740 12179 tablet_service.cc:1467] Processing CreateTablet for tablet e7732ff4fa2744db94d0f8cb888de8d8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ec56bc8cfbc04f26b92384c442eac5b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:39.711061 12179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7732ff4fa2744db94d0f8cb888de8d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:39.713954 12047 tablet_service.cc:1467] Processing CreateTablet for tablet 439f0129f4bd49b7b582be22be83e0b4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ec56bc8cfbc04f26b92384c442eac5b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:39.714291 12047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 439f0129f4bd49b7b582be22be83e0b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:39.716063 12965 tablet_bootstrap.cc:492] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a: Bootstrap starting.
I20240502 14:24:39.716187 12966 tablet_bootstrap.cc:492] T 439f0129f4bd49b7b582be22be83e0b4 P 0e41abe35d8f40a2a01953eea1c5d092: Bootstrap starting.
I20240502 14:24:39.717592 12966 tablet_bootstrap.cc:654] T 439f0129f4bd49b7b582be22be83e0b4 P 0e41abe35d8f40a2a01953eea1c5d092: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:39.718233 12965 tablet_bootstrap.cc:654] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:39.722115 12966 tablet_bootstrap.cc:492] T 439f0129f4bd49b7b582be22be83e0b4 P 0e41abe35d8f40a2a01953eea1c5d092: No bootstrap required, opened a new log
I20240502 14:24:39.722290 12966 ts_tablet_manager.cc:1367] T 439f0129f4bd49b7b582be22be83e0b4 P 0e41abe35d8f40a2a01953eea1c5d092: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.003s
I20240502 14:24:39.723134 12966 raft_consensus.cc:348] T 439f0129f4bd49b7b582be22be83e0b4 P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } }
I20240502 14:24:39.723443 12966 raft_consensus.cc:374] T 439f0129f4bd49b7b582be22be83e0b4 P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:39.723578 12966 raft_consensus.cc:729] T 439f0129f4bd49b7b582be22be83e0b4 P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e41abe35d8f40a2a01953eea1c5d092, State: Initialized, Role: FOLLOWER
I20240502 14:24:39.723887 12966 consensus_queue.cc:260] T 439f0129f4bd49b7b582be22be83e0b4 P 0e41abe35d8f40a2a01953eea1c5d092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } }
I20240502 14:24:39.725858 12970 tablet_bootstrap.cc:492] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105: Bootstrap starting.
I20240502 14:24:39.726475 12966 ts_tablet_manager.cc:1398] T 439f0129f4bd49b7b582be22be83e0b4 P 0e41abe35d8f40a2a01953eea1c5d092: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20240502 14:24:39.727016 12965 tablet_bootstrap.cc:492] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a: No bootstrap required, opened a new log
I20240502 14:24:39.727144 12965 ts_tablet_manager.cc:1367] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240502 14:24:39.727267 12970 tablet_bootstrap.cc:654] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:39.727890 12965 raft_consensus.cc:348] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:39.728103 12965 raft_consensus.cc:374] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:39.728230 12965 raft_consensus.cc:729] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36e66c7b1a03458b89a91c56a57df72a, State: Initialized, Role: FOLLOWER
I20240502 14:24:39.728428 12965 consensus_queue.cc:260] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:39.728891 12965 ts_tablet_manager.cc:1398] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:24:39.729030 12970 tablet_bootstrap.cc:492] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105: No bootstrap required, opened a new log
I20240502 14:24:39.729149 12970 ts_tablet_manager.cc:1367] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:24:39.729200 12965 tablet_bootstrap.cc:492] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a: Bootstrap starting.
I20240502 14:24:39.729804 12970 raft_consensus.cc:348] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:39.729979 12970 raft_consensus.cc:374] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:39.730108 12970 raft_consensus.cc:729] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9748c575342401bb80683c7e2c4e105, State: Initialized, Role: FOLLOWER
I20240502 14:24:39.730355 12970 consensus_queue.cc:260] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:39.730489 12965 tablet_bootstrap.cc:654] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:39.730957 12970 ts_tablet_manager.cc:1398] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:24:39.731561 12970 tablet_bootstrap.cc:492] T 003642773a724534847246de230ea34b P d9748c575342401bb80683c7e2c4e105: Bootstrap starting.
I20240502 14:24:39.732380 12965 tablet_bootstrap.cc:492] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a: No bootstrap required, opened a new log
I20240502 14:24:39.732544 12965 ts_tablet_manager.cc:1367] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240502 14:24:39.732965 12970 tablet_bootstrap.cc:654] T 003642773a724534847246de230ea34b P d9748c575342401bb80683c7e2c4e105: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:39.733403 12965 raft_consensus.cc:348] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:39.733626 12965 raft_consensus.cc:374] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:39.733708 12965 raft_consensus.cc:729] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36e66c7b1a03458b89a91c56a57df72a, State: Initialized, Role: FOLLOWER
I20240502 14:24:39.733978 12965 consensus_queue.cc:260] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:39.734548 12965 ts_tablet_manager.cc:1398] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:24:39.734861 12965 tablet_bootstrap.cc:492] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a: Bootstrap starting.
I20240502 14:24:39.736593 12965 tablet_bootstrap.cc:654] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:39.738360 12965 tablet_bootstrap.cc:492] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a: No bootstrap required, opened a new log
I20240502 14:24:39.738463 12965 ts_tablet_manager.cc:1367] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240502 14:24:39.738464 12970 tablet_bootstrap.cc:492] T 003642773a724534847246de230ea34b P d9748c575342401bb80683c7e2c4e105: No bootstrap required, opened a new log
I20240502 14:24:39.738608 12970 ts_tablet_manager.cc:1367] T 003642773a724534847246de230ea34b P d9748c575342401bb80683c7e2c4e105: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240502 14:24:39.738973 12965 raft_consensus.cc:348] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } }
I20240502 14:24:39.739087 12965 raft_consensus.cc:374] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:39.739161 12965 raft_consensus.cc:729] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36e66c7b1a03458b89a91c56a57df72a, State: Initialized, Role: FOLLOWER
I20240502 14:24:39.739121 12970 raft_consensus.cc:348] T 003642773a724534847246de230ea34b P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:39.739337 12970 raft_consensus.cc:374] T 003642773a724534847246de230ea34b P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:39.739338 12965 consensus_queue.cc:260] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } }
I20240502 14:24:39.739406 12970 raft_consensus.cc:729] T 003642773a724534847246de230ea34b P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9748c575342401bb80683c7e2c4e105, State: Initialized, Role: FOLLOWER
I20240502 14:24:39.739650 12970 consensus_queue.cc:260] T 003642773a724534847246de230ea34b P d9748c575342401bb80683c7e2c4e105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:39.739735 12965 ts_tablet_manager.cc:1398] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:24:39.740043 12970 ts_tablet_manager.cc:1398] T 003642773a724534847246de230ea34b P d9748c575342401bb80683c7e2c4e105: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:24:39.821463 12833 raft_consensus.cc:482] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:24:39.821678 12833 raft_consensus.cc:504] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:39.822176 12833 leader_election.cc:288] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9748c575342401bb80683c7e2c4e105 (127.5.107.67:45923)
I20240502 14:24:39.822698 12199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7732ff4fa2744db94d0f8cb888de8d8" candidate_uuid: "36e66c7b1a03458b89a91c56a57df72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9748c575342401bb80683c7e2c4e105" is_pre_election: true
I20240502 14:24:39.822911 12199 raft_consensus.cc:2454] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36e66c7b1a03458b89a91c56a57df72a in term 0.
I20240502 14:24:39.823323 11999 leader_election.cc:302] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 36e66c7b1a03458b89a91c56a57df72a, d9748c575342401bb80683c7e2c4e105; no voters:
I20240502 14:24:39.823596 12833 raft_consensus.cc:2781] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:24:39.823686 12833 raft_consensus.cc:482] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:24:39.823763 12833 raft_consensus.cc:3037] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:39.826735 12833 raft_consensus.cc:504] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:39.827082 12833 leader_election.cc:288] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 election: Requested vote from peers d9748c575342401bb80683c7e2c4e105 (127.5.107.67:45923)
I20240502 14:24:39.827491 12199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7732ff4fa2744db94d0f8cb888de8d8" candidate_uuid: "36e66c7b1a03458b89a91c56a57df72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9748c575342401bb80683c7e2c4e105"
I20240502 14:24:39.827641 12199 raft_consensus.cc:3037] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:39.829231 12199 raft_consensus.cc:2454] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36e66c7b1a03458b89a91c56a57df72a in term 1.
I20240502 14:24:39.829527 11999 leader_election.cc:302] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 36e66c7b1a03458b89a91c56a57df72a, d9748c575342401bb80683c7e2c4e105; no voters:
I20240502 14:24:39.829761 12833 raft_consensus.cc:2781] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:24:39.830145 12833 raft_consensus.cc:686] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Becoming Leader. State: Replica: 36e66c7b1a03458b89a91c56a57df72a, State: Running, Role: LEADER
I20240502 14:24:39.830402 12833 consensus_queue.cc:237] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:39.833258 11792 catalog_manager.cc:5484] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a reported cstate change: term changed from 0 to 1, leader changed from <none> to 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66). New cstate: current_term: 1 leader_uuid: "36e66c7b1a03458b89a91c56a57df72a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } }
I20240502 14:24:39.900492 12833 raft_consensus.cc:482] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:24:39.900643 12833 raft_consensus.cc:504] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:39.901213 12833 leader_election.cc:288] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9748c575342401bb80683c7e2c4e105 (127.5.107.67:45923)
I20240502 14:24:39.901674 12199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003642773a724534847246de230ea34b" candidate_uuid: "36e66c7b1a03458b89a91c56a57df72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9748c575342401bb80683c7e2c4e105" is_pre_election: true
I20240502 14:24:39.901916 12199 raft_consensus.cc:2454] T 003642773a724534847246de230ea34b P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36e66c7b1a03458b89a91c56a57df72a in term 0.
I20240502 14:24:39.902261 11999 leader_election.cc:302] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 36e66c7b1a03458b89a91c56a57df72a, d9748c575342401bb80683c7e2c4e105; no voters:
I20240502 14:24:39.902489 12833 raft_consensus.cc:2781] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:24:39.902601 12833 raft_consensus.cc:482] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:24:39.902683 12833 raft_consensus.cc:3037] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:39.904462 12833 raft_consensus.cc:504] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:39.904793 12833 leader_election.cc:288] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 election: Requested vote from peers d9748c575342401bb80683c7e2c4e105 (127.5.107.67:45923)
I20240502 14:24:39.905167 12199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "003642773a724534847246de230ea34b" candidate_uuid: "36e66c7b1a03458b89a91c56a57df72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9748c575342401bb80683c7e2c4e105"
I20240502 14:24:39.905330 12199 raft_consensus.cc:3037] T 003642773a724534847246de230ea34b P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:39.906916 12199 raft_consensus.cc:2454] T 003642773a724534847246de230ea34b P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36e66c7b1a03458b89a91c56a57df72a in term 1.
I20240502 14:24:39.907227 11999 leader_election.cc:302] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 36e66c7b1a03458b89a91c56a57df72a, d9748c575342401bb80683c7e2c4e105; no voters:
I20240502 14:24:39.907475 12833 raft_consensus.cc:2781] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:24:39.907603 12833 raft_consensus.cc:686] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Becoming Leader. State: Replica: 36e66c7b1a03458b89a91c56a57df72a, State: Running, Role: LEADER
I20240502 14:24:39.907816 12833 consensus_queue.cc:237] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:39.909844 11792 catalog_manager.cc:5484] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a reported cstate change: term changed from 0 to 1, leader changed from <none> to 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66). New cstate: current_term: 1 leader_uuid: "36e66c7b1a03458b89a91c56a57df72a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } }
I20240502 14:24:39.987967 12833 raft_consensus.cc:482] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:24:39.988148 12833 raft_consensus.cc:504] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } }
I20240502 14:24:39.988611 12833 leader_election.cc:288] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871)
I20240502 14:24:39.989235 11934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "439f0129f4bd49b7b582be22be83e0b4" candidate_uuid: "36e66c7b1a03458b89a91c56a57df72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e41abe35d8f40a2a01953eea1c5d092" is_pre_election: true
I20240502 14:24:39.989421 11934 raft_consensus.cc:2454] T 439f0129f4bd49b7b582be22be83e0b4 P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36e66c7b1a03458b89a91c56a57df72a in term 0.
I20240502 14:24:39.989806 12000 leader_election.cc:302] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e41abe35d8f40a2a01953eea1c5d092, 36e66c7b1a03458b89a91c56a57df72a; no voters:
I20240502 14:24:39.990067 12833 raft_consensus.cc:2781] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:24:39.990140 12833 raft_consensus.cc:482] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:24:39.990209 12833 raft_consensus.cc:3037] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:39.991858 12833 raft_consensus.cc:504] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } }
I20240502 14:24:39.992210 12833 leader_election.cc:288] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 election: Requested vote from peers 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871)
I20240502 14:24:39.992795 11934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "439f0129f4bd49b7b582be22be83e0b4" candidate_uuid: "36e66c7b1a03458b89a91c56a57df72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e41abe35d8f40a2a01953eea1c5d092"
I20240502 14:24:39.992954 11934 raft_consensus.cc:3037] T 439f0129f4bd49b7b582be22be83e0b4 P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:39.994679 11934 raft_consensus.cc:2454] T 439f0129f4bd49b7b582be22be83e0b4 P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36e66c7b1a03458b89a91c56a57df72a in term 1.
I20240502 14:24:39.995020 12000 leader_election.cc:302] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e41abe35d8f40a2a01953eea1c5d092, 36e66c7b1a03458b89a91c56a57df72a; no voters:
I20240502 14:24:39.995249 12833 raft_consensus.cc:2781] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:24:39.995380 12833 raft_consensus.cc:686] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Becoming Leader. State: Replica: 36e66c7b1a03458b89a91c56a57df72a, State: Running, Role: LEADER
I20240502 14:24:39.995652 12833 consensus_queue.cc:237] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } }
I20240502 14:24:39.997746 11792 catalog_manager.cc:5484] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a reported cstate change: term changed from 0 to 1, leader changed from <none> to 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66). New cstate: current_term: 1 leader_uuid: "36e66c7b1a03458b89a91c56a57df72a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } }
I20240502 14:24:40.266819 12833 consensus_queue.cc:1035] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:24:40.299962 12980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:24:40.300235 12980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:24:40.306679 12980 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 14:24:40.420055 11934 raft_consensus.cc:1261] T 439f0129f4bd49b7b582be22be83e0b4 P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Refusing update from remote peer 36e66c7b1a03458b89a91c56a57df72a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:24:40.420619 12199 raft_consensus.cc:1261] T 003642773a724534847246de230ea34b P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Refusing update from remote peer 36e66c7b1a03458b89a91c56a57df72a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:24:40.421638 12833 consensus_queue.cc:1035] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:24:40.421913 13013 consensus_queue.cc:1035] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:24:40.473392 12983 mvcc.cc:205] Tried to move back new op lower bound from 7023246870198358016 to 7023246868103647232. Current Snapshot: MvccSnapshot[applied={T|T < 7023246870178705408}]
INSERT report
rows total: 2000
time total: 146.228 ms
time per row: 0.073114 ms
I20240502 14:24:40.502465 13016 mvcc.cc:205] Tried to move back new op lower bound from 7023246870200958976 to 7023246868463403008. Current Snapshot: MvccSnapshot[applied={T|T < 7023246870200958976}]
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:24:41.435673 11786 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60480:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:24:41.443962 12179 tablet_service.cc:1467] Processing CreateTablet for tablet 877f199681534650bc1d247cf7608f5f (DEFAULT_TABLE table=rebalance_test_table_3 [id=423ad8739bd34e329c45d60dce8f9687]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:41.444449 12179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 877f199681534650bc1d247cf7608f5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:41.444471 11916 tablet_service.cc:1467] Processing CreateTablet for tablet 71d8679988d0444c9e0434d7e6f142e8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=423ad8739bd34e329c45d60dce8f9687]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:41.444912 11916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71d8679988d0444c9e0434d7e6f142e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:41.445132 12178 tablet_service.cc:1467] Processing CreateTablet for tablet 71d8679988d0444c9e0434d7e6f142e8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=423ad8739bd34e329c45d60dce8f9687]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:41.445253 11915 tablet_service.cc:1467] Processing CreateTablet for tablet 2f0a86c3cb6c48258f4d2f916b192c9c (DEFAULT_TABLE table=rebalance_test_table_3 [id=423ad8739bd34e329c45d60dce8f9687]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:41.445492 12178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71d8679988d0444c9e0434d7e6f142e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:41.446552 11915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f0a86c3cb6c48258f4d2f916b192c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:41.449510 11914 tablet_service.cc:1467] Processing CreateTablet for tablet 877f199681534650bc1d247cf7608f5f (DEFAULT_TABLE table=rebalance_test_table_3 [id=423ad8739bd34e329c45d60dce8f9687]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:41.449846 11914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 877f199681534650bc1d247cf7608f5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:41.454020 12047 tablet_service.cc:1467] Processing CreateTablet for tablet 2f0a86c3cb6c48258f4d2f916b192c9c (DEFAULT_TABLE table=rebalance_test_table_3 [id=423ad8739bd34e329c45d60dce8f9687]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:41.454365 12047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f0a86c3cb6c48258f4d2f916b192c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:41.459801 13034 tablet_bootstrap.cc:492] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a: Bootstrap starting.
I20240502 14:24:41.460692 13032 tablet_bootstrap.cc:492] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105: Bootstrap starting.
I20240502 14:24:41.461480 13034 tablet_bootstrap.cc:654] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:41.463701 13032 tablet_bootstrap.cc:654] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:41.464776 13033 tablet_bootstrap.cc:492] T 71d8679988d0444c9e0434d7e6f142e8 P 0e41abe35d8f40a2a01953eea1c5d092: Bootstrap starting.
I20240502 14:24:41.465622 13034 tablet_bootstrap.cc:492] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a: No bootstrap required, opened a new log
I20240502 14:24:41.465704 13032 tablet_bootstrap.cc:492] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105: No bootstrap required, opened a new log
I20240502 14:24:41.465745 13034 ts_tablet_manager.cc:1367] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 14:24:41.465834 13032 ts_tablet_manager.cc:1367] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:24:41.466285 13033 tablet_bootstrap.cc:654] T 71d8679988d0444c9e0434d7e6f142e8 P 0e41abe35d8f40a2a01953eea1c5d092: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:41.466331 13034 raft_consensus.cc:348] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:41.466495 13034 raft_consensus.cc:374] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:41.466584 13034 raft_consensus.cc:729] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36e66c7b1a03458b89a91c56a57df72a, State: Initialized, Role: FOLLOWER
I20240502 14:24:41.466562 13032 raft_consensus.cc:348] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:41.466754 13032 raft_consensus.cc:374] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:41.466856 13032 raft_consensus.cc:729] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9748c575342401bb80683c7e2c4e105, State: Initialized, Role: FOLLOWER
I20240502 14:24:41.466786 13034 consensus_queue.cc:260] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:41.467056 13032 consensus_queue.cc:260] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:41.467240 13034 ts_tablet_manager.cc:1398] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:24:41.468325 13032 ts_tablet_manager.cc:1398] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:24:41.468670 13032 tablet_bootstrap.cc:492] T 877f199681534650bc1d247cf7608f5f P d9748c575342401bb80683c7e2c4e105: Bootstrap starting.
I20240502 14:24:41.469569 13033 tablet_bootstrap.cc:492] T 71d8679988d0444c9e0434d7e6f142e8 P 0e41abe35d8f40a2a01953eea1c5d092: No bootstrap required, opened a new log
I20240502 14:24:41.469717 13033 ts_tablet_manager.cc:1367] T 71d8679988d0444c9e0434d7e6f142e8 P 0e41abe35d8f40a2a01953eea1c5d092: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:24:41.470461 13033 raft_consensus.cc:348] T 71d8679988d0444c9e0434d7e6f142e8 P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:41.470675 13033 raft_consensus.cc:374] T 71d8679988d0444c9e0434d7e6f142e8 P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:41.470772 13032 tablet_bootstrap.cc:654] T 877f199681534650bc1d247cf7608f5f P d9748c575342401bb80683c7e2c4e105: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:41.470785 13033 raft_consensus.cc:729] T 71d8679988d0444c9e0434d7e6f142e8 P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e41abe35d8f40a2a01953eea1c5d092, State: Initialized, Role: FOLLOWER
I20240502 14:24:41.471021 13033 consensus_queue.cc:260] T 71d8679988d0444c9e0434d7e6f142e8 P 0e41abe35d8f40a2a01953eea1c5d092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:41.473745 13033 ts_tablet_manager.cc:1398] T 71d8679988d0444c9e0434d7e6f142e8 P 0e41abe35d8f40a2a01953eea1c5d092: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:24:41.474074 13033 tablet_bootstrap.cc:492] T 2f0a86c3cb6c48258f4d2f916b192c9c P 0e41abe35d8f40a2a01953eea1c5d092: Bootstrap starting.
I20240502 14:24:41.475543 13033 tablet_bootstrap.cc:654] T 2f0a86c3cb6c48258f4d2f916b192c9c P 0e41abe35d8f40a2a01953eea1c5d092: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:41.477200 13033 tablet_bootstrap.cc:492] T 2f0a86c3cb6c48258f4d2f916b192c9c P 0e41abe35d8f40a2a01953eea1c5d092: No bootstrap required, opened a new log
I20240502 14:24:41.477334 13033 ts_tablet_manager.cc:1367] T 2f0a86c3cb6c48258f4d2f916b192c9c P 0e41abe35d8f40a2a01953eea1c5d092: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:24:41.477751 13032 tablet_bootstrap.cc:492] T 877f199681534650bc1d247cf7608f5f P d9748c575342401bb80683c7e2c4e105: No bootstrap required, opened a new log
I20240502 14:24:41.477880 13032 ts_tablet_manager.cc:1367] T 877f199681534650bc1d247cf7608f5f P d9748c575342401bb80683c7e2c4e105: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20240502 14:24:41.477921 13033 raft_consensus.cc:348] T 2f0a86c3cb6c48258f4d2f916b192c9c P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:41.478088 13033 raft_consensus.cc:374] T 2f0a86c3cb6c48258f4d2f916b192c9c P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:41.478155 13033 raft_consensus.cc:729] T 2f0a86c3cb6c48258f4d2f916b192c9c P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e41abe35d8f40a2a01953eea1c5d092, State: Initialized, Role: FOLLOWER
I20240502 14:24:41.478358 13033 consensus_queue.cc:260] T 2f0a86c3cb6c48258f4d2f916b192c9c P 0e41abe35d8f40a2a01953eea1c5d092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:41.478468 13032 raft_consensus.cc:348] T 877f199681534650bc1d247cf7608f5f P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } }
I20240502 14:24:41.478713 13032 raft_consensus.cc:374] T 877f199681534650bc1d247cf7608f5f P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:41.478793 13033 ts_tablet_manager.cc:1398] T 2f0a86c3cb6c48258f4d2f916b192c9c P 0e41abe35d8f40a2a01953eea1c5d092: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:24:41.478798 13032 raft_consensus.cc:729] T 877f199681534650bc1d247cf7608f5f P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9748c575342401bb80683c7e2c4e105, State: Initialized, Role: FOLLOWER
I20240502 14:24:41.479028 13033 tablet_bootstrap.cc:492] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092: Bootstrap starting.
I20240502 14:24:41.479004 13032 consensus_queue.cc:260] T 877f199681534650bc1d247cf7608f5f P d9748c575342401bb80683c7e2c4e105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } }
I20240502 14:24:41.479629 13032 ts_tablet_manager.cc:1398] T 877f199681534650bc1d247cf7608f5f P d9748c575342401bb80683c7e2c4e105: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:24:41.480336 13033 tablet_bootstrap.cc:654] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:41.482167 13033 tablet_bootstrap.cc:492] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092: No bootstrap required, opened a new log
I20240502 14:24:41.482280 13033 ts_tablet_manager.cc:1367] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:24:41.482841 13033 raft_consensus.cc:348] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } }
I20240502 14:24:41.483000 13033 raft_consensus.cc:374] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:41.483103 13033 raft_consensus.cc:729] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e41abe35d8f40a2a01953eea1c5d092, State: Initialized, Role: FOLLOWER
I20240502 14:24:41.483307 13033 consensus_queue.cc:260] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } }
I20240502 14:24:41.483705 13033 ts_tablet_manager.cc:1398] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 14:24:41.486034 12975 raft_consensus.cc:482] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:24:41.486157 12975 raft_consensus.cc:504] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:41.486582 12975 leader_election.cc:288] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871)
I20240502 14:24:41.487026 11936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f0a86c3cb6c48258f4d2f916b192c9c" candidate_uuid: "36e66c7b1a03458b89a91c56a57df72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e41abe35d8f40a2a01953eea1c5d092" is_pre_election: true
I20240502 14:24:41.487191 11936 raft_consensus.cc:2454] T 2f0a86c3cb6c48258f4d2f916b192c9c P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36e66c7b1a03458b89a91c56a57df72a in term 0.
I20240502 14:24:41.487545 12000 leader_election.cc:302] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e41abe35d8f40a2a01953eea1c5d092, 36e66c7b1a03458b89a91c56a57df72a; no voters:
I20240502 14:24:41.487766 12975 raft_consensus.cc:2781] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:24:41.487879 12975 raft_consensus.cc:482] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:24:41.487967 12975 raft_consensus.cc:3037] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:41.489724 12975 raft_consensus.cc:504] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:41.490069 12975 leader_election.cc:288] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 election: Requested vote from peers 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871)
I20240502 14:24:41.490495 11936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f0a86c3cb6c48258f4d2f916b192c9c" candidate_uuid: "36e66c7b1a03458b89a91c56a57df72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e41abe35d8f40a2a01953eea1c5d092"
I20240502 14:24:41.490643 11936 raft_consensus.cc:3037] T 2f0a86c3cb6c48258f4d2f916b192c9c P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:41.492326 11936 raft_consensus.cc:2454] T 2f0a86c3cb6c48258f4d2f916b192c9c P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36e66c7b1a03458b89a91c56a57df72a in term 1.
I20240502 14:24:41.492619 12000 leader_election.cc:302] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e41abe35d8f40a2a01953eea1c5d092, 36e66c7b1a03458b89a91c56a57df72a; no voters:
I20240502 14:24:41.492911 12975 raft_consensus.cc:2781] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:24:41.493255 12975 raft_consensus.cc:686] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Becoming Leader. State: Replica: 36e66c7b1a03458b89a91c56a57df72a, State: Running, Role: LEADER
I20240502 14:24:41.493484 12975 consensus_queue.cc:237] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:41.495291 11786 catalog_manager.cc:5484] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a reported cstate change: term changed from 0 to 1, leader changed from <none> to 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66). New cstate: current_term: 1 leader_uuid: "36e66c7b1a03458b89a91c56a57df72a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } }
I20240502 14:24:41.568482 12895 raft_consensus.cc:482] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:24:41.568675 12895 raft_consensus.cc:504] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:41.569113 12895 leader_election.cc:288] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871)
I20240502 14:24:41.569568 11936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71d8679988d0444c9e0434d7e6f142e8" candidate_uuid: "d9748c575342401bb80683c7e2c4e105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e41abe35d8f40a2a01953eea1c5d092" is_pre_election: true
I20240502 14:24:41.569782 11936 raft_consensus.cc:2454] T 71d8679988d0444c9e0434d7e6f142e8 P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9748c575342401bb80683c7e2c4e105 in term 0.
I20240502 14:24:41.570120 12133 leader_election.cc:302] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e41abe35d8f40a2a01953eea1c5d092, d9748c575342401bb80683c7e2c4e105; no voters:
I20240502 14:24:41.570406 12895 raft_consensus.cc:2781] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:24:41.570482 12895 raft_consensus.cc:482] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:24:41.570572 12895 raft_consensus.cc:3037] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:41.572430 12895 raft_consensus.cc:504] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:41.572890 12895 leader_election.cc:288] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105 [CANDIDATE]: Term 1 election: Requested vote from peers 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871)
I20240502 14:24:41.573416 11936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71d8679988d0444c9e0434d7e6f142e8" candidate_uuid: "d9748c575342401bb80683c7e2c4e105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e41abe35d8f40a2a01953eea1c5d092"
I20240502 14:24:41.573611 11936 raft_consensus.cc:3037] T 71d8679988d0444c9e0434d7e6f142e8 P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:41.575465 11936 raft_consensus.cc:2454] T 71d8679988d0444c9e0434d7e6f142e8 P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9748c575342401bb80683c7e2c4e105 in term 1.
I20240502 14:24:41.575785 12133 leader_election.cc:302] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e41abe35d8f40a2a01953eea1c5d092, d9748c575342401bb80683c7e2c4e105; no voters:
I20240502 14:24:41.576004 12895 raft_consensus.cc:2781] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:24:41.576296 12895 raft_consensus.cc:686] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105 [term 1 LEADER]: Becoming Leader. State: Replica: d9748c575342401bb80683c7e2c4e105, State: Running, Role: LEADER
I20240502 14:24:41.576498 12895 consensus_queue.cc:237] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:41.578461 11792 catalog_manager.cc:5484] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105 reported cstate change: term changed from 0 to 1, leader changed from <none> to d9748c575342401bb80683c7e2c4e105 (127.5.107.67). New cstate: current_term: 1 leader_uuid: "d9748c575342401bb80683c7e2c4e105" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } }
I20240502 14:24:41.627830 13038 raft_consensus.cc:482] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:24:41.627977 13038 raft_consensus.cc:504] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } }
I20240502 14:24:41.628564 13038 leader_election.cc:288] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9748c575342401bb80683c7e2c4e105 (127.5.107.67:45923)
I20240502 14:24:41.633749 12199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "877f199681534650bc1d247cf7608f5f" candidate_uuid: "0e41abe35d8f40a2a01953eea1c5d092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9748c575342401bb80683c7e2c4e105" is_pre_election: true
I20240502 14:24:41.633955 12199 raft_consensus.cc:2454] T 877f199681534650bc1d247cf7608f5f P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e41abe35d8f40a2a01953eea1c5d092 in term 0.
I20240502 14:24:41.634374 11868 leader_election.cc:302] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e41abe35d8f40a2a01953eea1c5d092, d9748c575342401bb80683c7e2c4e105; no voters:
I20240502 14:24:41.634619 13038 raft_consensus.cc:2781] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:24:41.634717 13038 raft_consensus.cc:482] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:24:41.634778 13038 raft_consensus.cc:3037] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:41.636443 13038 raft_consensus.cc:504] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } }
I20240502 14:24:41.636801 13038 leader_election.cc:288] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [CANDIDATE]: Term 1 election: Requested vote from peers d9748c575342401bb80683c7e2c4e105 (127.5.107.67:45923)
I20240502 14:24:41.637173 12199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "877f199681534650bc1d247cf7608f5f" candidate_uuid: "0e41abe35d8f40a2a01953eea1c5d092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9748c575342401bb80683c7e2c4e105"
I20240502 14:24:41.637336 12199 raft_consensus.cc:3037] T 877f199681534650bc1d247cf7608f5f P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:41.638809 12199 raft_consensus.cc:2454] T 877f199681534650bc1d247cf7608f5f P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e41abe35d8f40a2a01953eea1c5d092 in term 1.
I20240502 14:24:41.639104 11868 leader_election.cc:302] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e41abe35d8f40a2a01953eea1c5d092, d9748c575342401bb80683c7e2c4e105; no voters:
I20240502 14:24:41.639317 13038 raft_consensus.cc:2781] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:24:41.639631 13038 raft_consensus.cc:686] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 LEADER]: Becoming Leader. State: Replica: 0e41abe35d8f40a2a01953eea1c5d092, State: Running, Role: LEADER
I20240502 14:24:41.640106 13038 consensus_queue.cc:237] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } }
I20240502 14:24:41.642910 11792 catalog_manager.cc:5484] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65). New cstate: current_term: 1 leader_uuid: "0e41abe35d8f40a2a01953eea1c5d092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } }
W20240502 14:24:41.982223 13050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:24:41.982465 13050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:24:41.988816 13050 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 14:24:42.072139 13042 consensus_queue.cc:1035] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:24:42.125483 12895 consensus_queue.cc:1035] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:24:42.156502 12199 raft_consensus.cc:1261] T 877f199681534650bc1d247cf7608f5f P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Refusing update from remote peer 0e41abe35d8f40a2a01953eea1c5d092: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:24:42.157419 13038 consensus_queue.cc:1035] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 195.351 ms
time per row: 0.0976754 ms
I20240502 14:24:42.316555 11792 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60480:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:24:42.325244 12047 tablet_service.cc:1467] Processing CreateTablet for tablet d55e54737408455fadf1a1b4041edd0a (DEFAULT_TABLE table=rebalance_test_table_4 [id=cc8c51baff5f4081aca1cf7908b85057]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:42.325387 12178 tablet_service.cc:1467] Processing CreateTablet for tablet d55e54737408455fadf1a1b4041edd0a (DEFAULT_TABLE table=rebalance_test_table_4 [id=cc8c51baff5f4081aca1cf7908b85057]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:42.325734 12047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55e54737408455fadf1a1b4041edd0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:42.325824 12178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55e54737408455fadf1a1b4041edd0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:42.326013 12179 tablet_service.cc:1467] Processing CreateTablet for tablet 1427194e32d545f785df3e45137aa42e (DEFAULT_TABLE table=rebalance_test_table_4 [id=cc8c51baff5f4081aca1cf7908b85057]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:42.326406 12179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1427194e32d545f785df3e45137aa42e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:42.326776 11916 tablet_service.cc:1467] Processing CreateTablet for tablet f61f17c159094ff59d935ed0ef6381b8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=cc8c51baff5f4081aca1cf7908b85057]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:42.327224 11916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f61f17c159094ff59d935ed0ef6381b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:42.330227 12046 tablet_service.cc:1467] Processing CreateTablet for tablet 1427194e32d545f785df3e45137aa42e (DEFAULT_TABLE table=rebalance_test_table_4 [id=cc8c51baff5f4081aca1cf7908b85057]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:42.330552 12046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1427194e32d545f785df3e45137aa42e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:42.333142 13108 tablet_bootstrap.cc:492] T d55e54737408455fadf1a1b4041edd0a P d9748c575342401bb80683c7e2c4e105: Bootstrap starting.
I20240502 14:24:42.333489 12045 tablet_service.cc:1467] Processing CreateTablet for tablet f61f17c159094ff59d935ed0ef6381b8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=cc8c51baff5f4081aca1cf7908b85057]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:24:42.333743 12045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f61f17c159094ff59d935ed0ef6381b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:42.335422 13108 tablet_bootstrap.cc:654] T d55e54737408455fadf1a1b4041edd0a P d9748c575342401bb80683c7e2c4e105: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:42.337704 13108 tablet_bootstrap.cc:492] T d55e54737408455fadf1a1b4041edd0a P d9748c575342401bb80683c7e2c4e105: No bootstrap required, opened a new log
I20240502 14:24:42.337934 13108 ts_tablet_manager.cc:1367] T d55e54737408455fadf1a1b4041edd0a P d9748c575342401bb80683c7e2c4e105: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20240502 14:24:42.338919 13108 raft_consensus.cc:348] T d55e54737408455fadf1a1b4041edd0a P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:42.339217 13108 raft_consensus.cc:374] T d55e54737408455fadf1a1b4041edd0a P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:42.339360 13108 raft_consensus.cc:729] T d55e54737408455fadf1a1b4041edd0a P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9748c575342401bb80683c7e2c4e105, State: Initialized, Role: FOLLOWER
I20240502 14:24:42.339618 13108 consensus_queue.cc:260] T d55e54737408455fadf1a1b4041edd0a P d9748c575342401bb80683c7e2c4e105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:42.340355 13108 ts_tablet_manager.cc:1398] T d55e54737408455fadf1a1b4041edd0a P d9748c575342401bb80683c7e2c4e105: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:24:42.340776 13108 tablet_bootstrap.cc:492] T 1427194e32d545f785df3e45137aa42e P d9748c575342401bb80683c7e2c4e105: Bootstrap starting.
I20240502 14:24:42.342742 13108 tablet_bootstrap.cc:654] T 1427194e32d545f785df3e45137aa42e P d9748c575342401bb80683c7e2c4e105: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:42.346298 13109 tablet_bootstrap.cc:492] T f61f17c159094ff59d935ed0ef6381b8 P 0e41abe35d8f40a2a01953eea1c5d092: Bootstrap starting.
I20240502 14:24:42.346529 13111 tablet_bootstrap.cc:492] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a: Bootstrap starting.
I20240502 14:24:42.346961 13108 tablet_bootstrap.cc:492] T 1427194e32d545f785df3e45137aa42e P d9748c575342401bb80683c7e2c4e105: No bootstrap required, opened a new log
I20240502 14:24:42.347102 13108 ts_tablet_manager.cc:1367] T 1427194e32d545f785df3e45137aa42e P d9748c575342401bb80683c7e2c4e105: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20240502 14:24:42.347738 13109 tablet_bootstrap.cc:654] T f61f17c159094ff59d935ed0ef6381b8 P 0e41abe35d8f40a2a01953eea1c5d092: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:42.347808 13108 raft_consensus.cc:348] T 1427194e32d545f785df3e45137aa42e P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:42.348067 13108 raft_consensus.cc:374] T 1427194e32d545f785df3e45137aa42e P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:42.348119 13111 tablet_bootstrap.cc:654] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:42.348170 13108 raft_consensus.cc:729] T 1427194e32d545f785df3e45137aa42e P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9748c575342401bb80683c7e2c4e105, State: Initialized, Role: FOLLOWER
I20240502 14:24:42.348435 13108 consensus_queue.cc:260] T 1427194e32d545f785df3e45137aa42e P d9748c575342401bb80683c7e2c4e105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:42.350232 13111 tablet_bootstrap.cc:492] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a: No bootstrap required, opened a new log
I20240502 14:24:42.350314 13108 ts_tablet_manager.cc:1398] T 1427194e32d545f785df3e45137aa42e P d9748c575342401bb80683c7e2c4e105: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:24:42.350355 13111 ts_tablet_manager.cc:1367] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 14:24:42.350494 13109 tablet_bootstrap.cc:492] T f61f17c159094ff59d935ed0ef6381b8 P 0e41abe35d8f40a2a01953eea1c5d092: No bootstrap required, opened a new log
I20240502 14:24:42.350625 13109 ts_tablet_manager.cc:1367] T f61f17c159094ff59d935ed0ef6381b8 P 0e41abe35d8f40a2a01953eea1c5d092: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:24:42.351054 13111 raft_consensus.cc:348] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:42.351228 13111 raft_consensus.cc:374] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:42.351315 13111 raft_consensus.cc:729] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36e66c7b1a03458b89a91c56a57df72a, State: Initialized, Role: FOLLOWER
I20240502 14:24:42.351276 13109 raft_consensus.cc:348] T f61f17c159094ff59d935ed0ef6381b8 P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:42.351449 13109 raft_consensus.cc:374] T f61f17c159094ff59d935ed0ef6381b8 P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:42.351539 13109 raft_consensus.cc:729] T f61f17c159094ff59d935ed0ef6381b8 P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e41abe35d8f40a2a01953eea1c5d092, State: Initialized, Role: FOLLOWER
I20240502 14:24:42.351536 13111 consensus_queue.cc:260] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:42.351749 13109 consensus_queue.cc:260] T f61f17c159094ff59d935ed0ef6381b8 P 0e41abe35d8f40a2a01953eea1c5d092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:42.352087 13111 ts_tablet_manager.cc:1398] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:24:42.352329 13111 tablet_bootstrap.cc:492] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a: Bootstrap starting.
I20240502 14:24:42.352550 13109 ts_tablet_manager.cc:1398] T f61f17c159094ff59d935ed0ef6381b8 P 0e41abe35d8f40a2a01953eea1c5d092: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:24:42.353670 13111 tablet_bootstrap.cc:654] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:42.356074 13111 tablet_bootstrap.cc:492] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a: No bootstrap required, opened a new log
I20240502 14:24:42.356408 13111 ts_tablet_manager.cc:1367] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:24:42.357034 13111 raft_consensus.cc:348] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:42.357237 13111 raft_consensus.cc:374] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:42.357307 13111 raft_consensus.cc:729] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36e66c7b1a03458b89a91c56a57df72a, State: Initialized, Role: FOLLOWER
I20240502 14:24:42.357512 13111 consensus_queue.cc:260] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:42.358021 13111 ts_tablet_manager.cc:1398] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:24:42.358251 13111 tablet_bootstrap.cc:492] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a: Bootstrap starting.
I20240502 14:24:42.359452 13111 tablet_bootstrap.cc:654] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a: Neither blocks nor log segments found. Creating new log.
I20240502 14:24:42.360980 13111 tablet_bootstrap.cc:492] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a: No bootstrap required, opened a new log
I20240502 14:24:42.361078 13111 ts_tablet_manager.cc:1367] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:24:42.361585 13111 raft_consensus.cc:348] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:42.361721 13111 raft_consensus.cc:374] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:24:42.361783 13111 raft_consensus.cc:729] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36e66c7b1a03458b89a91c56a57df72a, State: Initialized, Role: FOLLOWER
I20240502 14:24:42.361945 13111 consensus_queue.cc:260] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:42.362300 13111 ts_tablet_manager.cc:1398] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:24:42.432276 13042 raft_consensus.cc:482] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:24:42.432495 13042 raft_consensus.cc:504] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:42.433143 13042 leader_election.cc:288] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9748c575342401bb80683c7e2c4e105 (127.5.107.67:45923)
I20240502 14:24:42.433688 12199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55e54737408455fadf1a1b4041edd0a" candidate_uuid: "36e66c7b1a03458b89a91c56a57df72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9748c575342401bb80683c7e2c4e105" is_pre_election: true
I20240502 14:24:42.433943 12199 raft_consensus.cc:2454] T d55e54737408455fadf1a1b4041edd0a P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36e66c7b1a03458b89a91c56a57df72a in term 0.
I20240502 14:24:42.434350 11999 leader_election.cc:302] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 36e66c7b1a03458b89a91c56a57df72a, d9748c575342401bb80683c7e2c4e105; no voters:
I20240502 14:24:42.434617 13042 raft_consensus.cc:2781] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:24:42.434700 13042 raft_consensus.cc:482] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:24:42.434800 13042 raft_consensus.cc:3037] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:42.436743 13042 raft_consensus.cc:504] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:42.437259 13042 leader_election.cc:288] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 election: Requested vote from peers d9748c575342401bb80683c7e2c4e105 (127.5.107.67:45923)
I20240502 14:24:42.437695 12199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55e54737408455fadf1a1b4041edd0a" candidate_uuid: "36e66c7b1a03458b89a91c56a57df72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9748c575342401bb80683c7e2c4e105"
I20240502 14:24:42.437857 12199 raft_consensus.cc:3037] T d55e54737408455fadf1a1b4041edd0a P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:42.439504 12199 raft_consensus.cc:2454] T d55e54737408455fadf1a1b4041edd0a P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36e66c7b1a03458b89a91c56a57df72a in term 1.
I20240502 14:24:42.439817 11999 leader_election.cc:302] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 36e66c7b1a03458b89a91c56a57df72a, d9748c575342401bb80683c7e2c4e105; no voters:
I20240502 14:24:42.440076 13042 raft_consensus.cc:2781] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:24:42.440223 13042 raft_consensus.cc:686] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Becoming Leader. State: Replica: 36e66c7b1a03458b89a91c56a57df72a, State: Running, Role: LEADER
I20240502 14:24:42.440470 13042 consensus_queue.cc:237] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:42.443207 11792 catalog_manager.cc:5484] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a reported cstate change: term changed from 0 to 1, leader changed from <none> to 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66). New cstate: current_term: 1 leader_uuid: "36e66c7b1a03458b89a91c56a57df72a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } }
I20240502 14:24:42.529811 13042 raft_consensus.cc:482] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:24:42.529958 13042 raft_consensus.cc:504] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:42.530454 13042 leader_election.cc:288] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871)
I20240502 14:24:42.530884 11936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61f17c159094ff59d935ed0ef6381b8" candidate_uuid: "36e66c7b1a03458b89a91c56a57df72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e41abe35d8f40a2a01953eea1c5d092" is_pre_election: true
I20240502 14:24:42.531075 11936 raft_consensus.cc:2454] T f61f17c159094ff59d935ed0ef6381b8 P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36e66c7b1a03458b89a91c56a57df72a in term 0.
I20240502 14:24:42.531395 12000 leader_election.cc:302] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e41abe35d8f40a2a01953eea1c5d092, 36e66c7b1a03458b89a91c56a57df72a; no voters:
I20240502 14:24:42.531625 13042 raft_consensus.cc:2781] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:24:42.531721 13042 raft_consensus.cc:482] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:24:42.531781 13042 raft_consensus.cc:3037] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:42.533720 13042 raft_consensus.cc:504] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:42.534066 13042 leader_election.cc:288] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 election: Requested vote from peers 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871)
I20240502 14:24:42.534555 11936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61f17c159094ff59d935ed0ef6381b8" candidate_uuid: "36e66c7b1a03458b89a91c56a57df72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e41abe35d8f40a2a01953eea1c5d092"
I20240502 14:24:42.534734 11936 raft_consensus.cc:3037] T f61f17c159094ff59d935ed0ef6381b8 P 0e41abe35d8f40a2a01953eea1c5d092 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:42.536365 11936 raft_consensus.cc:2454] T f61f17c159094ff59d935ed0ef6381b8 P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36e66c7b1a03458b89a91c56a57df72a in term 1.
I20240502 14:24:42.536680 12000 leader_election.cc:302] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e41abe35d8f40a2a01953eea1c5d092, 36e66c7b1a03458b89a91c56a57df72a; no voters:
I20240502 14:24:42.536936 13042 raft_consensus.cc:2781] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:24:42.537053 13042 raft_consensus.cc:686] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Becoming Leader. State: Replica: 36e66c7b1a03458b89a91c56a57df72a, State: Running, Role: LEADER
I20240502 14:24:42.537312 13042 consensus_queue.cc:237] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } }
I20240502 14:24:42.539566 11792 catalog_manager.cc:5484] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a reported cstate change: term changed from 0 to 1, leader changed from <none> to 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66). New cstate: current_term: 1 leader_uuid: "36e66c7b1a03458b89a91c56a57df72a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } }
I20240502 14:24:42.552575 13042 raft_consensus.cc:482] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:24:42.552754 13042 raft_consensus.cc:504] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:42.553148 13042 leader_election.cc:288] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9748c575342401bb80683c7e2c4e105 (127.5.107.67:45923)
I20240502 14:24:42.553678 12199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1427194e32d545f785df3e45137aa42e" candidate_uuid: "36e66c7b1a03458b89a91c56a57df72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9748c575342401bb80683c7e2c4e105" is_pre_election: true
I20240502 14:24:42.553900 12199 raft_consensus.cc:2454] T 1427194e32d545f785df3e45137aa42e P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36e66c7b1a03458b89a91c56a57df72a in term 0.
I20240502 14:24:42.554242 11999 leader_election.cc:302] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 36e66c7b1a03458b89a91c56a57df72a, d9748c575342401bb80683c7e2c4e105; no voters:
I20240502 14:24:42.554458 13042 raft_consensus.cc:2781] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:24:42.554569 13042 raft_consensus.cc:482] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:24:42.554670 13042 raft_consensus.cc:3037] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:42.556208 13042 raft_consensus.cc:504] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:42.556551 13042 leader_election.cc:288] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 election: Requested vote from peers d9748c575342401bb80683c7e2c4e105 (127.5.107.67:45923)
I20240502 14:24:42.556968 12199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1427194e32d545f785df3e45137aa42e" candidate_uuid: "36e66c7b1a03458b89a91c56a57df72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9748c575342401bb80683c7e2c4e105"
I20240502 14:24:42.557096 12199 raft_consensus.cc:3037] T 1427194e32d545f785df3e45137aa42e P d9748c575342401bb80683c7e2c4e105 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:24:42.558775 12199 raft_consensus.cc:2454] T 1427194e32d545f785df3e45137aa42e P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36e66c7b1a03458b89a91c56a57df72a in term 1.
I20240502 14:24:42.559062 11999 leader_election.cc:302] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 36e66c7b1a03458b89a91c56a57df72a, d9748c575342401bb80683c7e2c4e105; no voters:
I20240502 14:24:42.559269 13042 raft_consensus.cc:2781] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:24:42.559430 13042 raft_consensus.cc:686] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Becoming Leader. State: Replica: 36e66c7b1a03458b89a91c56a57df72a, State: Running, Role: LEADER
I20240502 14:24:42.559693 13042 consensus_queue.cc:237] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } }
I20240502 14:24:42.562440 11792 catalog_manager.cc:5484] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a reported cstate change: term changed from 0 to 1, leader changed from <none> to 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66). New cstate: current_term: 1 leader_uuid: "36e66c7b1a03458b89a91c56a57df72a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } }
W20240502 14:24:42.866044 13121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:24:42.866297 13121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:24:42.872578 13121 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 14:24:42.940186 13042 consensus_queue.cc:1035] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:24:42.995688 13042 consensus_queue.cc:1035] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:24:43.018246 11936 raft_consensus.cc:1261] T f61f17c159094ff59d935ed0ef6381b8 P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Refusing update from remote peer 36e66c7b1a03458b89a91c56a57df72a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:24:43.018800 13156 consensus_queue.cc:1035] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 175.305 ms
time per row: 0.0876523 ms
I20240502 14:24:43.173873 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.68:37273
--local_ip_for_outbound_sockets=127.5.107.68
--tserver_master_addrs=127.5.107.126:36087
--webserver_port=36281
--webserver_interface=127.5.107.68
--builtin_ntp_servers=127.5.107.84:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 14:24:43.481068 13169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:24:43.481346 13169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:24:43.481436 13169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:24:43.487769 13169 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:24:43.487857 13169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:24:43.487957 13169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.68
I20240502 14:24:43.496229 13169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.68:37273
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.5.107.68
--webserver_port=36281
--tserver_master_addrs=127.5.107.126:36087
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.68
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:24:43.497339 13169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:24:43.498831 13169 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:24:43.509056 13174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:43.509665 13175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:43.510227 13177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:24:43.510859 13169 server_base.cc:1034] running on GCE node
I20240502 14:24:43.511294 13169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:24:43.512041 13169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:24:43.513255 13169 hybrid_clock.cc:648] HybridClock initialized: now 1714659883513195 us; error 39 us; skew 500 ppm
I20240502 14:24:43.513553 13169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:24:43.515480 13169 webserver.cc:460] Webserver started at http://127.5.107.68:36281/ using document root <none> and password file <none>
I20240502 14:24:43.516077 13169 fs_manager.cc:359] Metadata directory not provided
I20240502 14:24:43.516183 13169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:24:43.520503 13169 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20240502 14:24:43.523022 13183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:43.524252 13169 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 14:24:43.524380 13169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "93c82e87d8af478292ffe1a3d3986049"
format_stamp: "Formatted at 2024-05-02 14:24:30 on dist-test-slave-2cw1"
I20240502 14:24:43.525060 13169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:24:43.536902 13169 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:24:43.537535 13169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:24:43.537770 13169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:24:43.538502 13169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:24:43.540145 13169 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:24:43.540241 13169 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:43.540361 13169 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:24:43.540416 13169 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:43.578285 13169 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.68:37273
I20240502 14:24:43.578320 13296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.68:37273 every 8 connection(s)
I20240502 14:24:43.579686 13169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:24:43.589828 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 13169
I20240502 14:24:43.590517 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.69:33193
--local_ip_for_outbound_sockets=127.5.107.69
--tserver_master_addrs=127.5.107.126:36087
--webserver_port=33211
--webserver_interface=127.5.107.69
--builtin_ntp_servers=127.5.107.84:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:24:43.592113 13297 heartbeater.cc:347] Connected to a master server at 127.5.107.126:36087
I20240502 14:24:43.592430 13297 heartbeater.cc:464] Registering TS with master...
I20240502 14:24:43.593344 13297 heartbeater.cc:511] Master 127.5.107.126:36087 requested a full tablet report, sending...
I20240502 14:24:43.594971 11792 ts_manager.cc:194] Re-registered known tserver with Master: 93c82e87d8af478292ffe1a3d3986049 (127.5.107.68:37273)
I20240502 14:24:43.595919 11792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.68:39919
I20240502 14:24:43.767377 13302 raft_consensus.cc:482] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:24:43.767547 13302 raft_consensus.cc:504] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:24:43.768035 13302 leader_election.cc:288] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 375c6d5bdf924866b69b0e6667732692 (127.5.107.70:33951)
W20240502 14:24:43.768826 11871 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:33951: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240502 14:24:43.769899 11871 leader_election.cc:334] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 375c6d5bdf924866b69b0e6667732692 (127.5.107.70:33951): Network error: Client connection negotiation failed: client connection to 127.5.107.70:33951: connect: Connection refused (error 111)
I20240502 14:24:43.769989 11871 leader_election.cc:302] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0e41abe35d8f40a2a01953eea1c5d092; no voters: 375c6d5bdf924866b69b0e6667732692
I20240502 14:24:43.770201 13302 raft_consensus.cc:2726] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:24:43.887406 13301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:24:43.887684 13301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:24:43.887846 13301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:24:43.894639 13301 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:24:43.894767 13301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:24:43.894908 13301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.69
I20240502 14:24:43.903046 13301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.69:33193
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.5.107.69
--webserver_port=33211
--tserver_master_addrs=127.5.107.126:36087
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.69
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:24:43.904309 13301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:24:43.905915 13301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:24:43.917295 13309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:43.917740 13312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:43.917894 13310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:24:43.918531 13301 server_base.cc:1034] running on GCE node
I20240502 14:24:43.918973 13301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:24:43.919687 13301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:24:43.920859 13301 hybrid_clock.cc:648] HybridClock initialized: now 1714659883920820 us; error 25 us; skew 500 ppm
I20240502 14:24:43.921149 13301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:24:43.925480 13301 webserver.cc:460] Webserver started at http://127.5.107.69:33211/ using document root <none> and password file <none>
I20240502 14:24:43.926133 13301 fs_manager.cc:359] Metadata directory not provided
I20240502 14:24:43.926254 13301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:24:43.934456 13301 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.005s sys 0.003s
I20240502 14:24:43.937764 13318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:43.938975 13301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:24:43.939090 13301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "723fb53f34c94c6aa5f7ee3a65711268"
format_stamp: "Formatted at 2024-05-02 14:24:30 on dist-test-slave-2cw1"
I20240502 14:24:43.939757 13301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:24:43.953207 13301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:24:43.953907 13301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:24:43.954180 13301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:24:43.955003 13301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:24:43.956270 13301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:24:43.956343 13301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:43.956462 13301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:24:43.956507 13301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:43.984207 13301 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.69:33193
I20240502 14:24:43.984292 13432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.69:33193 every 8 connection(s)
I20240502 14:24:43.985574 13301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:24:43.988468 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 13301
I20240502 14:24:43.989078 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.70:33951
--local_ip_for_outbound_sockets=127.5.107.70
--tserver_master_addrs=127.5.107.126:36087
--webserver_port=39797
--webserver_interface=127.5.107.70
--builtin_ntp_servers=127.5.107.84:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:24:43.998577 13433 heartbeater.cc:347] Connected to a master server at 127.5.107.126:36087
I20240502 14:24:43.998852 13433 heartbeater.cc:464] Registering TS with master...
I20240502 14:24:43.999475 13433 heartbeater.cc:511] Master 127.5.107.126:36087 requested a full tablet report, sending...
I20240502 14:24:44.000619 11792 ts_manager.cc:194] Re-registered known tserver with Master: 723fb53f34c94c6aa5f7ee3a65711268 (127.5.107.69:33193)
I20240502 14:24:44.001466 11792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.69:49547
W20240502 14:24:44.292799 13436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:24:44.293046 13436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:24:44.293233 13436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:24:44.299824 13436 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:24:44.299934 13436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:24:44.300055 13436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.70
I20240502 14:24:44.308378 13436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.70:33951
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.5.107.70
--webserver_port=39797
--tserver_master_addrs=127.5.107.126:36087
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.70
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:24:44.309563 13436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:24:44.311127 13436 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:24:44.321738 13444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:44.322486 13443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:44.322927 13446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:24:44.323275 13436 server_base.cc:1034] running on GCE node
I20240502 14:24:44.323820 13436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:24:44.324748 13436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:24:44.325934 13436 hybrid_clock.cc:648] HybridClock initialized: now 1714659884325893 us; error 44 us; skew 500 ppm
I20240502 14:24:44.326264 13436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:24:44.328261 13436 webserver.cc:460] Webserver started at http://127.5.107.70:39797/ using document root <none> and password file <none>
I20240502 14:24:44.328907 13436 fs_manager.cc:359] Metadata directory not provided
I20240502 14:24:44.328997 13436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:24:44.333889 13436 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:24:44.336314 13452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:44.337522 13436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:24:44.337674 13436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "375c6d5bdf924866b69b0e6667732692"
format_stamp: "Formatted at 2024-05-02 14:24:31 on dist-test-slave-2cw1"
I20240502 14:24:44.338454 13436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:24:44.362740 13436 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:24:44.363440 13436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:24:44.363721 13436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:24:44.364570 13436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:24:44.366302 13459 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240502 14:24:44.371323 13436 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 14:24:44.371428 13436 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20240502 14:24:44.371575 13436 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 14:24:44.374356 13436 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 14:24:44.374454 13436 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20240502 14:24:44.374794 13459 tablet_bootstrap.cc:492] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692: Bootstrap starting.
I20240502 14:24:44.394421 13459 log.cc:824] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692: Log is configured to *not* fsync() on all Append() calls
I20240502 14:24:44.398087 13459 tablet_bootstrap.cc:492] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:24:44.398746 13459 tablet_bootstrap.cc:492] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692: Bootstrap complete.
I20240502 14:24:44.399787 13459 ts_tablet_manager.cc:1367] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692: Time spent bootstrapping tablet: real 0.025s user 0.014s sys 0.009s
I20240502 14:24:44.410336 13459 raft_consensus.cc:348] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:24:44.411334 13459 raft_consensus.cc:729] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 375c6d5bdf924866b69b0e6667732692, State: Initialized, Role: FOLLOWER
I20240502 14:24:44.412024 13459 consensus_queue.cc:260] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:24:44.413563 13459 ts_tablet_manager.cc:1398] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692: Time spent starting tablet: real 0.014s user 0.011s sys 0.005s
I20240502 14:24:44.420774 13436 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.70:33951
I20240502 14:24:44.420804 13568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.70:33951 every 8 connection(s)
I20240502 14:24:44.422262 13436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data/info.pb
W20240502 14:24:44.423678 13570 tablet.cc:2369] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:24:44.427724 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 13436
I20240502 14:24:44.428507 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.71:33799
--local_ip_for_outbound_sockets=127.5.107.71
--tserver_master_addrs=127.5.107.126:36087
--webserver_port=38813
--webserver_interface=127.5.107.71
--builtin_ntp_servers=127.5.107.84:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:24:44.441572 13569 heartbeater.cc:347] Connected to a master server at 127.5.107.126:36087
I20240502 14:24:44.441825 13569 heartbeater.cc:464] Registering TS with master...
I20240502 14:24:44.442535 13569 heartbeater.cc:511] Master 127.5.107.126:36087 requested a full tablet report, sending...
I20240502 14:24:44.444782 11792 ts_manager.cc:194] Re-registered known tserver with Master: 375c6d5bdf924866b69b0e6667732692 (127.5.107.70:33951)
I20240502 14:24:44.445981 11792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.70:42205
I20240502 14:24:44.598259 13297 heartbeater.cc:503] Master 127.5.107.126:36087 was elected leader, sending a full tablet report...
W20240502 14:24:44.827752 13572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:24:44.828034 13572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:24:44.828161 13572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:24:44.834594 13572 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:24:44.834713 13572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:24:44.834849 13572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.71
I20240502 14:24:44.842512 13572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.71:33799
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.5.107.71
--webserver_port=38813
--tserver_master_addrs=127.5.107.126:36087
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.71
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:24:44.843675 13572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:24:44.845395 13572 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:24:44.856339 13579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:44.857107 13580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:24:44.857586 13582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:24:44.857959 13572 server_base.cc:1034] running on GCE node
I20240502 14:24:44.858500 13572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:24:44.859313 13572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:24:44.860482 13572 hybrid_clock.cc:648] HybridClock initialized: now 1714659884860432 us; error 43 us; skew 500 ppm
I20240502 14:24:44.860800 13572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:24:44.862892 13572 webserver.cc:460] Webserver started at http://127.5.107.71:38813/ using document root <none> and password file <none>
I20240502 14:24:44.863525 13572 fs_manager.cc:359] Metadata directory not provided
I20240502 14:24:44.863626 13572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:24:44.868295 13572 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20240502 14:24:44.870880 13588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:44.872013 13572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:24:44.872150 13572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "b0d1c78721a746e99d0d3056f1e31371"
format_stamp: "Formatted at 2024-05-02 14:24:31 on dist-test-slave-2cw1"
I20240502 14:24:44.872881 13572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:24:44.885756 13572 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:24:44.886408 13572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:24:44.886659 13572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:24:44.887476 13572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:24:44.888792 13572 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:24:44.888861 13572 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:44.888986 13572 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:24:44.889060 13572 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:24:44.917114 13572 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.71:33799
I20240502 14:24:44.917163 13702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.71:33799 every 8 connection(s)
I20240502 14:24:44.918397 13572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 14:24:44.925959 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 13572
I20240502 14:24:44.926288 5549 test_util.cc:268] Using random seed: 1566180648
I20240502 14:24:44.931468 13703 heartbeater.cc:347] Connected to a master server at 127.5.107.126:36087
I20240502 14:24:44.931721 13703 heartbeater.cc:464] Registering TS with master...
I20240502 14:24:44.932358 13703 heartbeater.cc:511] Master 127.5.107.126:36087 requested a full tablet report, sending...
I20240502 14:24:44.933521 11792 ts_manager.cc:194] Re-registered known tserver with Master: b0d1c78721a746e99d0d3056f1e31371 (127.5.107.71:33799)
I20240502 14:24:44.934340 11792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.71:51831
I20240502 14:24:44.949807 5549 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240502 14:24:44.950241 5549 test_util.cc:268] Using random seed: 1566204606
I20240502 14:24:44.964205 5549 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240502 14:24:44.965468 5549 test_util.cc:268] Using random seed: 1566219748
I20240502 14:24:44.990417 5549 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240502 14:24:44.994515 5549 test_util.cc:268] Using random seed: 1566248846
I20240502 14:24:45.004446 13433 heartbeater.cc:503] Master 127.5.107.126:36087 was elected leader, sending a full tablet report...
I20240502 14:24:45.049005 5549 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240502 14:24:45.063688 5549 test_util.cc:268] Using random seed: 1566318020
I20240502 14:24:45.171927 5549 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240502 14:24:45.448187 13569 heartbeater.cc:503] Master 127.5.107.126:36087 was elected leader, sending a full tablet report...
I20240502 14:24:45.936563 13703 heartbeater.cc:503] Master 127.5.107.126:36087 was elected leader, sending a full tablet report...
I20240502 14:24:46.028877 12045 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:24:46.056253 11916 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:24:46.072019 13501 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:46.085070 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:46.109210 13637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:46.132612 12178 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:24:46.133338 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:46.288645 13853 raft_consensus.cc:482] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:24:46.289077 13853 raft_consensus.cc:504] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:24:46.292543 13853 leader_election.cc:288] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871)
I20240502 14:24:46.406994 11936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f279550142c648d89b81feee9740f43e" candidate_uuid: "375c6d5bdf924866b69b0e6667732692" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0e41abe35d8f40a2a01953eea1c5d092" is_pre_election: true
I20240502 14:24:46.407454 11936 raft_consensus.cc:2454] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 375c6d5bdf924866b69b0e6667732692 in term 1.
I20240502 14:24:46.408218 13454 leader_election.cc:302] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e41abe35d8f40a2a01953eea1c5d092, 375c6d5bdf924866b69b0e6667732692; no voters:
I20240502 14:24:46.408787 13853 raft_consensus.cc:2781] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:24:46.408955 13853 raft_consensus.cc:482] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:24:46.409061 13853 raft_consensus.cc:3037] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:24:46.419375 13853 raft_consensus.cc:504] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:24:46.421607 13853 leader_election.cc:288] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [CANDIDATE]: Term 2 election: Requested vote from peers 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871)
I20240502 14:24:46.424068 11932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f279550142c648d89b81feee9740f43e" candidate_uuid: "375c6d5bdf924866b69b0e6667732692" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0e41abe35d8f40a2a01953eea1c5d092"
I20240502 14:24:46.424304 11932 raft_consensus.cc:3037] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:24:46.456095 11932 raft_consensus.cc:2454] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 375c6d5bdf924866b69b0e6667732692 in term 2.
I20240502 14:24:46.458127 13454 leader_election.cc:302] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e41abe35d8f40a2a01953eea1c5d092, 375c6d5bdf924866b69b0e6667732692; no voters:
I20240502 14:24:46.458453 13853 raft_consensus.cc:2781] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:24:46.542589 13853 raft_consensus.cc:686] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [term 2 LEADER]: Becoming Leader. State: Replica: 375c6d5bdf924866b69b0e6667732692, State: Running, Role: LEADER
I20240502 14:24:46.543334 13853 consensus_queue.cc:237] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:24:46.552023 11787 catalog_manager.cc:5484] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "375c6d5bdf924866b69b0e6667732692" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } } }
I20240502 14:24:46.930109 11935 raft_consensus.cc:1261] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [term 2 FOLLOWER]: Refusing update from remote peer 375c6d5bdf924866b69b0e6667732692: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:24:46.931803 13853 consensus_queue.cc:1035] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:24:46.972666 11916 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:24:47.104516 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:47.122936 13637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:47.129941 12045 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:24:47.140627 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:47.142954 12178 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:24:47.157244 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:24:47.562449 12068 consensus_queue.cc:237] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 1.51, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } }
I20240502 14:24:47.569142 11933 raft_consensus.cc:1261] T 439f0129f4bd49b7b582be22be83e0b4 P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Refusing update from remote peer 36e66c7b1a03458b89a91c56a57df72a: Log matching property violated. Preceding OpId in replica: term: 1 index: 51. Preceding OpId from leader: term: 1 index: 52. (index mismatch)
I20240502 14:24:47.575301 13801 consensus_queue.cc:1035] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20240502 14:24:47.584635 13781 raft_consensus.cc:2932] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Committing config change with OpId 1.52: config changed from index -1 to 52, VOTER 723fb53f34c94c6aa5f7ee3a65711268 (127.5.107.69) added. New config: { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } }
I20240502 14:24:47.588284 11935 raft_consensus.cc:2932] T 439f0129f4bd49b7b582be22be83e0b4 P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Committing config change with OpId 1.52: config changed from index -1 to 52, VOTER 723fb53f34c94c6aa5f7ee3a65711268 (127.5.107.69) added. New config: { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } }
W20240502 14:24:47.589447 11999 consensus_peers.cc:485] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a -> Peer 723fb53f34c94c6aa5f7ee3a65711268 (127.5.107.69:33193): Couldn't send request to peer 723fb53f34c94c6aa5f7ee3a65711268. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 439f0129f4bd49b7b582be22be83e0b4. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:24:47.594301 11787 catalog_manager.cc:5484] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a reported cstate change: config changed from index -1 to 52, VOTER 723fb53f34c94c6aa5f7ee3a65711268 (127.5.107.69) added. New cstate: current_term: 1 leader_uuid: "36e66c7b1a03458b89a91c56a57df72a" committed_config { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } }
I20240502 14:24:47.798514 12200 consensus_queue.cc:237] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 1.84, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } }
I20240502 14:24:47.802124 11935 raft_consensus.cc:1261] T 71d8679988d0444c9e0434d7e6f142e8 P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Refusing update from remote peer d9748c575342401bb80683c7e2c4e105: Log matching property violated. Preceding OpId in replica: term: 1 index: 84. Preceding OpId from leader: term: 1 index: 85. (index mismatch)
I20240502 14:24:47.803185 13929 consensus_queue.cc:1035] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20240502 14:24:47.814311 13854 raft_consensus.cc:2932] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105 [term 1 LEADER]: Committing config change with OpId 1.85: config changed from index -1 to 85, VOTER b0d1c78721a746e99d0d3056f1e31371 (127.5.107.71) added. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } }
I20240502 14:24:47.827023 11933 raft_consensus.cc:2932] T 71d8679988d0444c9e0434d7e6f142e8 P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Committing config change with OpId 1.85: config changed from index -1 to 85, VOTER b0d1c78721a746e99d0d3056f1e31371 (127.5.107.71) added. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } }
I20240502 14:24:47.828369 11788 catalog_manager.cc:5484] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105 reported cstate change: config changed from index -1 to 85, VOTER b0d1c78721a746e99d0d3056f1e31371 (127.5.107.71) added. New cstate: current_term: 1 leader_uuid: "d9748c575342401bb80683c7e2c4e105" committed_config { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } }
W20240502 14:24:47.877833 12134 consensus_peers.cc:485] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105 -> Peer b0d1c78721a746e99d0d3056f1e31371 (127.5.107.71:33799): Couldn't send request to peer b0d1c78721a746e99d0d3056f1e31371. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71d8679988d0444c9e0434d7e6f142e8. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:24:48.020789 13967 ts_tablet_manager.cc:902] T 439f0129f4bd49b7b582be22be83e0b4 P 723fb53f34c94c6aa5f7ee3a65711268: Initiating tablet copy from peer 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445)
I20240502 14:24:48.054936 13967 tablet_copy_client.cc:308] T 439f0129f4bd49b7b582be22be83e0b4 P 723fb53f34c94c6aa5f7ee3a65711268: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:35445
I20240502 14:24:48.082737 12088 tablet_copy_service.cc:140] P 36e66c7b1a03458b89a91c56a57df72a: Received BeginTabletCopySession request for tablet 439f0129f4bd49b7b582be22be83e0b4 from peer 723fb53f34c94c6aa5f7ee3a65711268 ({username='slave'} at 127.5.107.69:39951)
I20240502 14:24:48.083073 12088 tablet_copy_service.cc:161] P 36e66c7b1a03458b89a91c56a57df72a: Beginning new tablet copy session on tablet 439f0129f4bd49b7b582be22be83e0b4 from peer 723fb53f34c94c6aa5f7ee3a65711268 at {username='slave'} at 127.5.107.69:39951: session id = 723fb53f34c94c6aa5f7ee3a65711268-439f0129f4bd49b7b582be22be83e0b4
I20240502 14:24:48.087334 12088 tablet_copy_source_session.cc:205] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:24:48.095095 13967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 439f0129f4bd49b7b582be22be83e0b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:48.123262 13967 tablet_copy_client.cc:783] T 439f0129f4bd49b7b582be22be83e0b4 P 723fb53f34c94c6aa5f7ee3a65711268: tablet copy: Starting download of 0 data blocks...
I20240502 14:24:48.123574 13967 tablet_copy_client.cc:647] T 439f0129f4bd49b7b582be22be83e0b4 P 723fb53f34c94c6aa5f7ee3a65711268: tablet copy: Starting download of 1 WAL segments...
I20240502 14:24:48.127239 13967 tablet_copy_client.cc:523] T 439f0129f4bd49b7b582be22be83e0b4 P 723fb53f34c94c6aa5f7ee3a65711268: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:24:48.130898 13967 tablet_bootstrap.cc:492] T 439f0129f4bd49b7b582be22be83e0b4 P 723fb53f34c94c6aa5f7ee3a65711268: Bootstrap starting.
I20240502 14:24:48.155694 13967 log.cc:824] T 439f0129f4bd49b7b582be22be83e0b4 P 723fb53f34c94c6aa5f7ee3a65711268: Log is configured to *not* fsync() on all Append() calls
I20240502 14:24:48.168226 12068 consensus_queue.cc:237] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 1.87, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } }
I20240502 14:24:48.189107 11936 raft_consensus.cc:1261] T 4987edf6ce4d44d18162ff06e3f74e23 P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Refusing update from remote peer 36e66c7b1a03458b89a91c56a57df72a: Log matching property violated. Preceding OpId in replica: term: 1 index: 87. Preceding OpId from leader: term: 1 index: 88. (index mismatch)
I20240502 14:24:48.190843 13156 consensus_queue.cc:1035] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 86, Time since last communication: 0.000s
I20240502 14:24:48.199103 13156 raft_consensus.cc:2932] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Committing config change with OpId 1.88: config changed from index -1 to 88, VOTER 93c82e87d8af478292ffe1a3d3986049 (127.5.107.68) added. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } } }
I20240502 14:24:48.206210 11788 catalog_manager.cc:5484] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a reported cstate change: config changed from index -1 to 88, VOTER 93c82e87d8af478292ffe1a3d3986049 (127.5.107.68) added. New cstate: current_term: 1 leader_uuid: "36e66c7b1a03458b89a91c56a57df72a" committed_config { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } } }
I20240502 14:24:48.212487 11933 raft_consensus.cc:2932] T 4987edf6ce4d44d18162ff06e3f74e23 P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Committing config change with OpId 1.88: config changed from index -1 to 88, VOTER 93c82e87d8af478292ffe1a3d3986049 (127.5.107.68) added. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } } }
W20240502 14:24:48.247884 12002 consensus_peers.cc:485] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a -> Peer 93c82e87d8af478292ffe1a3d3986049 (127.5.107.68:37273): Couldn't send request to peer 93c82e87d8af478292ffe1a3d3986049. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4987edf6ce4d44d18162ff06e3f74e23. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:24:48.258800 13982 ts_tablet_manager.cc:902] T 71d8679988d0444c9e0434d7e6f142e8 P b0d1c78721a746e99d0d3056f1e31371: Initiating tablet copy from peer d9748c575342401bb80683c7e2c4e105 (127.5.107.67:45923)
I20240502 14:24:48.298741 13982 tablet_copy_client.cc:308] T 71d8679988d0444c9e0434d7e6f142e8 P b0d1c78721a746e99d0d3056f1e31371: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.67:45923
I20240502 14:24:48.323688 13967 tablet_bootstrap.cc:492] T 439f0129f4bd49b7b582be22be83e0b4 P 723fb53f34c94c6aa5f7ee3a65711268: Bootstrap replayed 1/1 log segments. Stats: ops{read=67 overwritten=0 applied=67 ignored=0} inserts{seen=739 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:24:48.324442 13967 tablet_bootstrap.cc:492] T 439f0129f4bd49b7b582be22be83e0b4 P 723fb53f34c94c6aa5f7ee3a65711268: Bootstrap complete.
I20240502 14:24:48.325121 13967 ts_tablet_manager.cc:1367] T 439f0129f4bd49b7b582be22be83e0b4 P 723fb53f34c94c6aa5f7ee3a65711268: Time spent bootstrapping tablet: real 0.194s user 0.119s sys 0.008s
I20240502 14:24:48.331835 13967 raft_consensus.cc:348] T 439f0129f4bd49b7b582be22be83e0b4 P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } }
I20240502 14:24:48.332391 13967 raft_consensus.cc:729] T 439f0129f4bd49b7b582be22be83e0b4 P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723fb53f34c94c6aa5f7ee3a65711268, State: Initialized, Role: FOLLOWER
I20240502 14:24:48.333242 13967 consensus_queue.cc:260] T 439f0129f4bd49b7b582be22be83e0b4 P 723fb53f34c94c6aa5f7ee3a65711268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 67, Last appended: 1.67, Last appended by leader: 67, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } }
I20240502 14:24:48.350041 12220 tablet_copy_service.cc:140] P d9748c575342401bb80683c7e2c4e105: Received BeginTabletCopySession request for tablet 71d8679988d0444c9e0434d7e6f142e8 from peer b0d1c78721a746e99d0d3056f1e31371 ({username='slave'} at 127.5.107.71:58851)
I20240502 14:24:48.350267 12220 tablet_copy_service.cc:161] P d9748c575342401bb80683c7e2c4e105: Beginning new tablet copy session on tablet 71d8679988d0444c9e0434d7e6f142e8 from peer b0d1c78721a746e99d0d3056f1e31371 at {username='slave'} at 127.5.107.71:58851: session id = b0d1c78721a746e99d0d3056f1e31371-71d8679988d0444c9e0434d7e6f142e8
I20240502 14:24:48.355937 12220 tablet_copy_source_session.cc:205] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:24:48.356649 13967 ts_tablet_manager.cc:1398] T 439f0129f4bd49b7b582be22be83e0b4 P 723fb53f34c94c6aa5f7ee3a65711268: Time spent starting tablet: real 0.031s user 0.006s sys 0.008s
I20240502 14:24:48.358412 12088 tablet_copy_service.cc:342] P 36e66c7b1a03458b89a91c56a57df72a: Request end of tablet copy session 723fb53f34c94c6aa5f7ee3a65711268-439f0129f4bd49b7b582be22be83e0b4 received from {username='slave'} at 127.5.107.69:39951
I20240502 14:24:48.358582 12088 tablet_copy_service.cc:433] P 36e66c7b1a03458b89a91c56a57df72a: ending tablet copy session 723fb53f34c94c6aa5f7ee3a65711268-439f0129f4bd49b7b582be22be83e0b4 on tablet 439f0129f4bd49b7b582be22be83e0b4 with peer 723fb53f34c94c6aa5f7ee3a65711268
I20240502 14:24:48.362432 13982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71d8679988d0444c9e0434d7e6f142e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:48.380898 13387 raft_consensus.cc:1203] T 439f0129f4bd49b7b582be22be83e0b4 P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.51->[1.52-1.72] Dedup: 1.67->[1.68-1.72]
I20240502 14:24:48.390836 13982 tablet_copy_client.cc:783] T 71d8679988d0444c9e0434d7e6f142e8 P b0d1c78721a746e99d0d3056f1e31371: tablet copy: Starting download of 0 data blocks...
I20240502 14:24:48.391463 13982 tablet_copy_client.cc:647] T 71d8679988d0444c9e0434d7e6f142e8 P b0d1c78721a746e99d0d3056f1e31371: tablet copy: Starting download of 1 WAL segments...
I20240502 14:24:48.396068 13982 tablet_copy_client.cc:523] T 71d8679988d0444c9e0434d7e6f142e8 P b0d1c78721a746e99d0d3056f1e31371: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:24:48.399909 13982 tablet_bootstrap.cc:492] T 71d8679988d0444c9e0434d7e6f142e8 P b0d1c78721a746e99d0d3056f1e31371: Bootstrap starting.
I20240502 14:24:48.440902 13982 log.cc:824] T 71d8679988d0444c9e0434d7e6f142e8 P b0d1c78721a746e99d0d3056f1e31371: Log is configured to *not* fsync() on all Append() calls
I20240502 14:24:48.540629 12068 consensus_queue.cc:237] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 1.80, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:24:48.543952 12200 raft_consensus.cc:1261] T 003642773a724534847246de230ea34b P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Refusing update from remote peer 36e66c7b1a03458b89a91c56a57df72a: Log matching property violated. Preceding OpId in replica: term: 1 index: 80. Preceding OpId from leader: term: 1 index: 81. (index mismatch)
I20240502 14:24:48.546723 13156 consensus_queue.cc:1035] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20240502 14:24:48.566771 13781 raft_consensus.cc:2932] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Committing config change with OpId 1.81: config changed from index -1 to 81, VOTER 375c6d5bdf924866b69b0e6667732692 (127.5.107.70) added. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } } }
I20240502 14:24:48.571179 12196 raft_consensus.cc:2932] T 003642773a724534847246de230ea34b P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Committing config change with OpId 1.81: config changed from index -1 to 81, VOTER 375c6d5bdf924866b69b0e6667732692 (127.5.107.70) added. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } } }
W20240502 14:24:48.575680 12002 consensus_peers.cc:485] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a -> Peer 375c6d5bdf924866b69b0e6667732692 (127.5.107.70:33951): Couldn't send request to peer 375c6d5bdf924866b69b0e6667732692. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 003642773a724534847246de230ea34b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:24:48.576700 11787 catalog_manager.cc:5484] T 003642773a724534847246de230ea34b P d9748c575342401bb80683c7e2c4e105 reported cstate change: config changed from index -1 to 81, VOTER 375c6d5bdf924866b69b0e6667732692 (127.5.107.70) added. New cstate: current_term: 1 leader_uuid: "36e66c7b1a03458b89a91c56a57df72a" committed_config { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } } }
I20240502 14:24:48.579244 14011 ts_tablet_manager.cc:902] T 4987edf6ce4d44d18162ff06e3f74e23 P 93c82e87d8af478292ffe1a3d3986049: Initiating tablet copy from peer 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445)
I20240502 14:24:48.603123 13982 tablet_bootstrap.cc:492] T 71d8679988d0444c9e0434d7e6f142e8 P b0d1c78721a746e99d0d3056f1e31371: Bootstrap replayed 1/1 log segments. Stats: ops{read=98 overwritten=0 applied=97 ignored=0} inserts{seen=728 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:24:48.603878 13982 tablet_bootstrap.cc:492] T 71d8679988d0444c9e0434d7e6f142e8 P b0d1c78721a746e99d0d3056f1e31371: Bootstrap complete.
I20240502 14:24:48.604432 13982 ts_tablet_manager.cc:1367] T 71d8679988d0444c9e0434d7e6f142e8 P b0d1c78721a746e99d0d3056f1e31371: Time spent bootstrapping tablet: real 0.205s user 0.160s sys 0.005s
I20240502 14:24:48.612428 13982 raft_consensus.cc:348] T 71d8679988d0444c9e0434d7e6f142e8 P b0d1c78721a746e99d0d3056f1e31371 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } }
I20240502 14:24:48.622614 13982 raft_consensus.cc:729] T 71d8679988d0444c9e0434d7e6f142e8 P b0d1c78721a746e99d0d3056f1e31371 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0d1c78721a746e99d0d3056f1e31371, State: Initialized, Role: FOLLOWER
I20240502 14:24:48.623658 13982 consensus_queue.cc:260] T 71d8679988d0444c9e0434d7e6f142e8 P b0d1c78721a746e99d0d3056f1e31371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 97, Last appended: 1.98, Last appended by leader: 98, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } }
I20240502 14:24:48.635020 13982 ts_tablet_manager.cc:1398] T 71d8679988d0444c9e0434d7e6f142e8 P b0d1c78721a746e99d0d3056f1e31371: Time spent starting tablet: real 0.030s user 0.023s sys 0.005s
I20240502 14:24:48.640511 12220 tablet_copy_service.cc:342] P d9748c575342401bb80683c7e2c4e105: Request end of tablet copy session b0d1c78721a746e99d0d3056f1e31371-71d8679988d0444c9e0434d7e6f142e8 received from {username='slave'} at 127.5.107.71:58851
I20240502 14:24:48.640714 12220 tablet_copy_service.cc:433] P d9748c575342401bb80683c7e2c4e105: ending tablet copy session b0d1c78721a746e99d0d3056f1e31371-71d8679988d0444c9e0434d7e6f142e8 on tablet 71d8679988d0444c9e0434d7e6f142e8 with peer b0d1c78721a746e99d0d3056f1e31371
I20240502 14:24:48.654163 14011 tablet_copy_client.cc:308] T 4987edf6ce4d44d18162ff06e3f74e23 P 93c82e87d8af478292ffe1a3d3986049: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:35445
I20240502 14:24:48.680104 12088 tablet_copy_service.cc:140] P 36e66c7b1a03458b89a91c56a57df72a: Received BeginTabletCopySession request for tablet 4987edf6ce4d44d18162ff06e3f74e23 from peer 93c82e87d8af478292ffe1a3d3986049 ({username='slave'} at 127.5.107.68:41265)
I20240502 14:24:48.680277 12088 tablet_copy_service.cc:161] P 36e66c7b1a03458b89a91c56a57df72a: Beginning new tablet copy session on tablet 4987edf6ce4d44d18162ff06e3f74e23 from peer 93c82e87d8af478292ffe1a3d3986049 at {username='slave'} at 127.5.107.68:41265: session id = 93c82e87d8af478292ffe1a3d3986049-4987edf6ce4d44d18162ff06e3f74e23
I20240502 14:24:48.683902 12088 tablet_copy_source_session.cc:205] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:24:48.689823 14011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4987edf6ce4d44d18162ff06e3f74e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:48.738245 14011 tablet_copy_client.cc:783] T 4987edf6ce4d44d18162ff06e3f74e23 P 93c82e87d8af478292ffe1a3d3986049: tablet copy: Starting download of 0 data blocks...
I20240502 14:24:48.738942 14011 tablet_copy_client.cc:647] T 4987edf6ce4d44d18162ff06e3f74e23 P 93c82e87d8af478292ffe1a3d3986049: tablet copy: Starting download of 1 WAL segments...
I20240502 14:24:48.745579 14011 tablet_copy_client.cc:523] T 4987edf6ce4d44d18162ff06e3f74e23 P 93c82e87d8af478292ffe1a3d3986049: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:24:48.750468 13657 raft_consensus.cc:1203] T 71d8679988d0444c9e0434d7e6f142e8 P b0d1c78721a746e99d0d3056f1e31371 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.84->[1.85-1.109] Dedup: 1.98->[1.99-1.109]
I20240502 14:24:48.750617 14011 tablet_bootstrap.cc:492] T 4987edf6ce4d44d18162ff06e3f74e23 P 93c82e87d8af478292ffe1a3d3986049: Bootstrap starting.
I20240502 14:24:48.792410 14011 log.cc:824] T 4987edf6ce4d44d18162ff06e3f74e23 P 93c82e87d8af478292ffe1a3d3986049: Log is configured to *not* fsync() on all Append() calls
I20240502 14:24:48.989794 12068 consensus_queue.cc:237] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 1.62, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } }
W20240502 14:24:48.992931 11999 consensus_peers.cc:485] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a -> Peer 723fb53f34c94c6aa5f7ee3a65711268 (127.5.107.69:33193): Couldn't send request to peer 723fb53f34c94c6aa5f7ee3a65711268. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 64e9d77a5b9343489db94f4988180213. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:24:48.995260 11933 raft_consensus.cc:1261] T 64e9d77a5b9343489db94f4988180213 P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Refusing update from remote peer 36e66c7b1a03458b89a91c56a57df72a: Log matching property violated. Preceding OpId in replica: term: 1 index: 62. Preceding OpId from leader: term: 1 index: 63. (index mismatch)
I20240502 14:24:48.996101 13156 consensus_queue.cc:1035] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20240502 14:24:48.999701 13156 raft_consensus.cc:2932] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Committing config change with OpId 1.63: config changed from index -1 to 63, VOTER 723fb53f34c94c6aa5f7ee3a65711268 (127.5.107.69) added. New config: { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } }
I20240502 14:24:49.005676 11788 catalog_manager.cc:5484] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a reported cstate change: config changed from index -1 to 63, VOTER 723fb53f34c94c6aa5f7ee3a65711268 (127.5.107.69) added. New cstate: current_term: 1 leader_uuid: "36e66c7b1a03458b89a91c56a57df72a" committed_config { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } }
I20240502 14:24:49.011286 11933 raft_consensus.cc:2932] T 64e9d77a5b9343489db94f4988180213 P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Committing config change with OpId 1.63: config changed from index -1 to 63, VOTER 723fb53f34c94c6aa5f7ee3a65711268 (127.5.107.69) added. New config: { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } }
I20240502 14:24:49.015229 14011 tablet_bootstrap.cc:492] T 4987edf6ce4d44d18162ff06e3f74e23 P 93c82e87d8af478292ffe1a3d3986049: Bootstrap replayed 1/1 log segments. Stats: ops{read=91 overwritten=0 applied=91 ignored=0} inserts{seen=763 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:24:49.015851 14011 tablet_bootstrap.cc:492] T 4987edf6ce4d44d18162ff06e3f74e23 P 93c82e87d8af478292ffe1a3d3986049: Bootstrap complete.
I20240502 14:24:49.016363 14011 ts_tablet_manager.cc:1367] T 4987edf6ce4d44d18162ff06e3f74e23 P 93c82e87d8af478292ffe1a3d3986049: Time spent bootstrapping tablet: real 0.266s user 0.147s sys 0.028s
I20240502 14:24:49.022231 14011 raft_consensus.cc:348] T 4987edf6ce4d44d18162ff06e3f74e23 P 93c82e87d8af478292ffe1a3d3986049 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } }
I20240502 14:24:49.022706 14011 raft_consensus.cc:729] T 4987edf6ce4d44d18162ff06e3f74e23 P 93c82e87d8af478292ffe1a3d3986049 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93c82e87d8af478292ffe1a3d3986049, State: Initialized, Role: FOLLOWER
I20240502 14:24:49.023581 14011 consensus_queue.cc:260] T 4987edf6ce4d44d18162ff06e3f74e23 P 93c82e87d8af478292ffe1a3d3986049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 91, Last appended: 1.91, Last appended by leader: 91, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } }
I20240502 14:24:49.157999 14011 ts_tablet_manager.cc:1398] T 4987edf6ce4d44d18162ff06e3f74e23 P 93c82e87d8af478292ffe1a3d3986049: Time spent starting tablet: real 0.142s user 0.062s sys 0.073s
I20240502 14:24:49.165876 12088 tablet_copy_service.cc:342] P 36e66c7b1a03458b89a91c56a57df72a: Request end of tablet copy session 93c82e87d8af478292ffe1a3d3986049-4987edf6ce4d44d18162ff06e3f74e23 received from {username='slave'} at 127.5.107.68:41265
I20240502 14:24:49.166023 12088 tablet_copy_service.cc:433] P 36e66c7b1a03458b89a91c56a57df72a: ending tablet copy session 93c82e87d8af478292ffe1a3d3986049-4987edf6ce4d44d18162ff06e3f74e23 on tablet 4987edf6ce4d44d18162ff06e3f74e23 with peer 93c82e87d8af478292ffe1a3d3986049
I20240502 14:24:49.170630 14051 ts_tablet_manager.cc:902] T 003642773a724534847246de230ea34b P 375c6d5bdf924866b69b0e6667732692: Initiating tablet copy from peer 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445)
I20240502 14:24:49.249303 14051 tablet_copy_client.cc:308] T 003642773a724534847246de230ea34b P 375c6d5bdf924866b69b0e6667732692: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:35445
I20240502 14:24:49.294744 12088 tablet_copy_service.cc:140] P 36e66c7b1a03458b89a91c56a57df72a: Received BeginTabletCopySession request for tablet 003642773a724534847246de230ea34b from peer 375c6d5bdf924866b69b0e6667732692 ({username='slave'} at 127.5.107.70:60345)
I20240502 14:24:49.294907 12088 tablet_copy_service.cc:161] P 36e66c7b1a03458b89a91c56a57df72a: Beginning new tablet copy session on tablet 003642773a724534847246de230ea34b from peer 375c6d5bdf924866b69b0e6667732692 at {username='slave'} at 127.5.107.70:60345: session id = 375c6d5bdf924866b69b0e6667732692-003642773a724534847246de230ea34b
I20240502 14:24:49.297472 12088 tablet_copy_source_session.cc:205] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:24:49.299899 14051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 003642773a724534847246de230ea34b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:49.308534 14051 tablet_copy_client.cc:783] T 003642773a724534847246de230ea34b P 375c6d5bdf924866b69b0e6667732692: tablet copy: Starting download of 0 data blocks...
I20240502 14:24:49.308862 14051 tablet_copy_client.cc:647] T 003642773a724534847246de230ea34b P 375c6d5bdf924866b69b0e6667732692: tablet copy: Starting download of 1 WAL segments...
I20240502 14:24:49.323194 14051 tablet_copy_client.cc:523] T 003642773a724534847246de230ea34b P 375c6d5bdf924866b69b0e6667732692: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:24:49.328487 14051 tablet_bootstrap.cc:492] T 003642773a724534847246de230ea34b P 375c6d5bdf924866b69b0e6667732692: Bootstrap starting.
I20240502 14:24:49.483084 12068 consensus_queue.cc:237] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 1.70, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } }
I20240502 14:24:49.489346 14076 ts_tablet_manager.cc:902] T 64e9d77a5b9343489db94f4988180213 P 723fb53f34c94c6aa5f7ee3a65711268: Initiating tablet copy from peer 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445)
I20240502 14:24:49.502836 12196 raft_consensus.cc:1261] T d55e54737408455fadf1a1b4041edd0a P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Refusing update from remote peer 36e66c7b1a03458b89a91c56a57df72a: Log matching property violated. Preceding OpId in replica: term: 1 index: 70. Preceding OpId from leader: term: 1 index: 72. (index mismatch)
I20240502 14:24:49.504868 13858 consensus_queue.cc:1035] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20240502 14:24:49.517282 12197 raft_consensus.cc:2932] T d55e54737408455fadf1a1b4041edd0a P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Committing config change with OpId 1.71: config changed from index -1 to 71, VOTER b0d1c78721a746e99d0d3056f1e31371 (127.5.107.71) added. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } }
I20240502 14:24:49.520100 13801 raft_consensus.cc:2932] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Committing config change with OpId 1.71: config changed from index -1 to 71, VOTER b0d1c78721a746e99d0d3056f1e31371 (127.5.107.71) added. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } }
I20240502 14:24:49.524163 11788 catalog_manager.cc:5484] T d55e54737408455fadf1a1b4041edd0a P d9748c575342401bb80683c7e2c4e105 reported cstate change: config changed from index -1 to 71, VOTER b0d1c78721a746e99d0d3056f1e31371 (127.5.107.71) added. New cstate: current_term: 1 leader_uuid: "36e66c7b1a03458b89a91c56a57df72a" committed_config { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } }
I20240502 14:24:49.544404 14076 tablet_copy_client.cc:308] T 64e9d77a5b9343489db94f4988180213 P 723fb53f34c94c6aa5f7ee3a65711268: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:35445
I20240502 14:24:49.545061 12088 tablet_copy_service.cc:140] P 36e66c7b1a03458b89a91c56a57df72a: Received BeginTabletCopySession request for tablet 64e9d77a5b9343489db94f4988180213 from peer 723fb53f34c94c6aa5f7ee3a65711268 ({username='slave'} at 127.5.107.69:39951)
I20240502 14:24:49.545238 12088 tablet_copy_service.cc:161] P 36e66c7b1a03458b89a91c56a57df72a: Beginning new tablet copy session on tablet 64e9d77a5b9343489db94f4988180213 from peer 723fb53f34c94c6aa5f7ee3a65711268 at {username='slave'} at 127.5.107.69:39951: session id = 723fb53f34c94c6aa5f7ee3a65711268-64e9d77a5b9343489db94f4988180213
I20240502 14:24:49.547791 12088 tablet_copy_source_session.cc:205] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:24:49.550268 14076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64e9d77a5b9343489db94f4988180213. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:49.565626 14076 tablet_copy_client.cc:783] T 64e9d77a5b9343489db94f4988180213 P 723fb53f34c94c6aa5f7ee3a65711268: tablet copy: Starting download of 0 data blocks...
I20240502 14:24:49.565874 14076 tablet_copy_client.cc:647] T 64e9d77a5b9343489db94f4988180213 P 723fb53f34c94c6aa5f7ee3a65711268: tablet copy: Starting download of 1 WAL segments...
I20240502 14:24:49.569432 14076 tablet_copy_client.cc:523] T 64e9d77a5b9343489db94f4988180213 P 723fb53f34c94c6aa5f7ee3a65711268: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:24:49.572785 14051 tablet_bootstrap.cc:492] T 003642773a724534847246de230ea34b P 375c6d5bdf924866b69b0e6667732692: Bootstrap replayed 1/1 log segments. Stats: ops{read=98 overwritten=0 applied=98 ignored=0} inserts{seen=783 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:24:49.573323 14051 tablet_bootstrap.cc:492] T 003642773a724534847246de230ea34b P 375c6d5bdf924866b69b0e6667732692: Bootstrap complete.
I20240502 14:24:49.573661 14051 ts_tablet_manager.cc:1367] T 003642773a724534847246de230ea34b P 375c6d5bdf924866b69b0e6667732692: Time spent bootstrapping tablet: real 0.245s user 0.158s sys 0.023s
I20240502 14:24:49.574806 14051 raft_consensus.cc:348] T 003642773a724534847246de230ea34b P 375c6d5bdf924866b69b0e6667732692 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:24:49.575074 14051 raft_consensus.cc:729] T 003642773a724534847246de230ea34b P 375c6d5bdf924866b69b0e6667732692 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 375c6d5bdf924866b69b0e6667732692, State: Initialized, Role: FOLLOWER
I20240502 14:24:49.575337 14051 consensus_queue.cc:260] T 003642773a724534847246de230ea34b P 375c6d5bdf924866b69b0e6667732692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 98, Last appended: 1.98, Last appended by leader: 98, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:24:49.575950 14051 ts_tablet_manager.cc:1398] T 003642773a724534847246de230ea34b P 375c6d5bdf924866b69b0e6667732692: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:24:49.581095 14076 tablet_bootstrap.cc:492] T 64e9d77a5b9343489db94f4988180213 P 723fb53f34c94c6aa5f7ee3a65711268: Bootstrap starting.
I20240502 14:24:49.582193 12088 tablet_copy_service.cc:342] P 36e66c7b1a03458b89a91c56a57df72a: Request end of tablet copy session 375c6d5bdf924866b69b0e6667732692-003642773a724534847246de230ea34b received from {username='slave'} at 127.5.107.70:60345
I20240502 14:24:49.582331 12088 tablet_copy_service.cc:433] P 36e66c7b1a03458b89a91c56a57df72a: ending tablet copy session 375c6d5bdf924866b69b0e6667732692-003642773a724534847246de230ea34b on tablet 003642773a724534847246de230ea34b with peer 375c6d5bdf924866b69b0e6667732692
I20240502 14:24:49.609808 13251 raft_consensus.cc:1203] T 4987edf6ce4d44d18162ff06e3f74e23 P 93c82e87d8af478292ffe1a3d3986049 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.87->[1.88-1.106] Dedup: 1.91->[1.92-1.106]
I20240502 14:24:49.627604 13522 raft_consensus.cc:1203] T 003642773a724534847246de230ea34b P 375c6d5bdf924866b69b0e6667732692 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.80->[1.81-1.108] Dedup: 1.98->[1.99-1.108]
W20240502 14:24:49.680661 12001 consensus_peers.cc:485] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a -> Peer b0d1c78721a746e99d0d3056f1e31371 (127.5.107.71:33799): Couldn't send request to peer b0d1c78721a746e99d0d3056f1e31371. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d55e54737408455fadf1a1b4041edd0a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:24:49.826583 14076 tablet_bootstrap.cc:492] T 64e9d77a5b9343489db94f4988180213 P 723fb53f34c94c6aa5f7ee3a65711268: Bootstrap replayed 1/1 log segments. Stats: ops{read=71 overwritten=0 applied=71 ignored=0} inserts{seen=743 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:24:49.827169 14076 tablet_bootstrap.cc:492] T 64e9d77a5b9343489db94f4988180213 P 723fb53f34c94c6aa5f7ee3a65711268: Bootstrap complete.
I20240502 14:24:49.827384 14076 ts_tablet_manager.cc:1367] T 64e9d77a5b9343489db94f4988180213 P 723fb53f34c94c6aa5f7ee3a65711268: Time spent bootstrapping tablet: real 0.246s user 0.118s sys 0.023s
I20240502 14:24:49.828338 14076 raft_consensus.cc:348] T 64e9d77a5b9343489db94f4988180213 P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } }
I20240502 14:24:49.828620 14076 raft_consensus.cc:729] T 64e9d77a5b9343489db94f4988180213 P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723fb53f34c94c6aa5f7ee3a65711268, State: Initialized, Role: FOLLOWER
I20240502 14:24:49.828871 14076 consensus_queue.cc:260] T 64e9d77a5b9343489db94f4988180213 P 723fb53f34c94c6aa5f7ee3a65711268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 71, Last appended: 1.71, Last appended by leader: 71, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } }
I20240502 14:24:49.829697 14076 ts_tablet_manager.cc:1398] T 64e9d77a5b9343489db94f4988180213 P 723fb53f34c94c6aa5f7ee3a65711268: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:24:49.830411 12088 tablet_copy_service.cc:342] P 36e66c7b1a03458b89a91c56a57df72a: Request end of tablet copy session 723fb53f34c94c6aa5f7ee3a65711268-64e9d77a5b9343489db94f4988180213 received from {username='slave'} at 127.5.107.69:39951
I20240502 14:24:49.830556 12088 tablet_copy_service.cc:433] P 36e66c7b1a03458b89a91c56a57df72a: ending tablet copy session 723fb53f34c94c6aa5f7ee3a65711268-64e9d77a5b9343489db94f4988180213 on tablet 64e9d77a5b9343489db94f4988180213 with peer 723fb53f34c94c6aa5f7ee3a65711268
I20240502 14:24:49.976727 13386 raft_consensus.cc:1203] T 64e9d77a5b9343489db94f4988180213 P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.62->[1.63-1.80] Dedup: 1.71->[1.72-1.80]
I20240502 14:24:49.995952 14117 ts_tablet_manager.cc:902] T d55e54737408455fadf1a1b4041edd0a P b0d1c78721a746e99d0d3056f1e31371: Initiating tablet copy from peer 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445)
I20240502 14:24:50.010066 12199 consensus_queue.cc:237] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 1.84, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } }
I20240502 14:24:50.016161 11933 raft_consensus.cc:1261] T 9069116816104c79819b25838836575b P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Refusing update from remote peer d9748c575342401bb80683c7e2c4e105: Log matching property violated. Preceding OpId in replica: term: 1 index: 84. Preceding OpId from leader: term: 1 index: 85. (index mismatch)
I20240502 14:24:50.027050 13854 consensus_queue.cc:1035] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
W20240502 14:24:50.037423 12135 consensus_peers.cc:485] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105 -> Peer 93c82e87d8af478292ffe1a3d3986049 (127.5.107.68:37273): Couldn't send request to peer 93c82e87d8af478292ffe1a3d3986049. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9069116816104c79819b25838836575b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:24:50.066458 13854 raft_consensus.cc:2932] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105 [term 1 LEADER]: Committing config change with OpId 1.85: config changed from index -1 to 85, VOTER 93c82e87d8af478292ffe1a3d3986049 (127.5.107.68) added. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } } }
I20240502 14:24:50.090389 11936 raft_consensus.cc:2932] T 9069116816104c79819b25838836575b P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Committing config change with OpId 1.85: config changed from index -1 to 85, VOTER 93c82e87d8af478292ffe1a3d3986049 (127.5.107.68) added. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } } }
I20240502 14:24:50.097229 11787 catalog_manager.cc:5484] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105 reported cstate change: config changed from index -1 to 85, VOTER 93c82e87d8af478292ffe1a3d3986049 (127.5.107.68) added. New cstate: current_term: 1 leader_uuid: "d9748c575342401bb80683c7e2c4e105" committed_config { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } } }
I20240502 14:24:50.112983 14117 tablet_copy_client.cc:308] T d55e54737408455fadf1a1b4041edd0a P b0d1c78721a746e99d0d3056f1e31371: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:35445
I20240502 14:24:50.187317 12088 tablet_copy_service.cc:140] P 36e66c7b1a03458b89a91c56a57df72a: Received BeginTabletCopySession request for tablet d55e54737408455fadf1a1b4041edd0a from peer b0d1c78721a746e99d0d3056f1e31371 ({username='slave'} at 127.5.107.71:39197)
I20240502 14:24:50.187497 12088 tablet_copy_service.cc:161] P 36e66c7b1a03458b89a91c56a57df72a: Beginning new tablet copy session on tablet d55e54737408455fadf1a1b4041edd0a from peer b0d1c78721a746e99d0d3056f1e31371 at {username='slave'} at 127.5.107.71:39197: session id = b0d1c78721a746e99d0d3056f1e31371-d55e54737408455fadf1a1b4041edd0a
I20240502 14:24:50.192571 12088 tablet_copy_source_session.cc:205] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:24:50.201371 14117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55e54737408455fadf1a1b4041edd0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:50.207760 14117 tablet_copy_client.cc:783] T d55e54737408455fadf1a1b4041edd0a P b0d1c78721a746e99d0d3056f1e31371: tablet copy: Starting download of 0 data blocks...
I20240502 14:24:50.208319 14117 tablet_copy_client.cc:647] T d55e54737408455fadf1a1b4041edd0a P b0d1c78721a746e99d0d3056f1e31371: tablet copy: Starting download of 1 WAL segments...
I20240502 14:24:50.235558 14117 tablet_copy_client.cc:523] T d55e54737408455fadf1a1b4041edd0a P b0d1c78721a746e99d0d3056f1e31371: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:24:50.238765 14117 tablet_bootstrap.cc:492] T d55e54737408455fadf1a1b4041edd0a P b0d1c78721a746e99d0d3056f1e31371: Bootstrap starting.
I20240502 14:24:50.438948 14117 tablet_bootstrap.cc:492] T d55e54737408455fadf1a1b4041edd0a P b0d1c78721a746e99d0d3056f1e31371: Bootstrap replayed 1/1 log segments. Stats: ops{read=84 overwritten=0 applied=83 ignored=0} inserts{seen=768 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:24:50.439580 14117 tablet_bootstrap.cc:492] T d55e54737408455fadf1a1b4041edd0a P b0d1c78721a746e99d0d3056f1e31371: Bootstrap complete.
I20240502 14:24:50.439973 14117 ts_tablet_manager.cc:1367] T d55e54737408455fadf1a1b4041edd0a P b0d1c78721a746e99d0d3056f1e31371: Time spent bootstrapping tablet: real 0.201s user 0.121s sys 0.013s
I20240502 14:24:50.441099 14117 raft_consensus.cc:348] T d55e54737408455fadf1a1b4041edd0a P b0d1c78721a746e99d0d3056f1e31371 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } }
I20240502 14:24:50.442184 14117 raft_consensus.cc:729] T d55e54737408455fadf1a1b4041edd0a P b0d1c78721a746e99d0d3056f1e31371 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0d1c78721a746e99d0d3056f1e31371, State: Initialized, Role: FOLLOWER
I20240502 14:24:50.442458 14117 consensus_queue.cc:260] T d55e54737408455fadf1a1b4041edd0a P b0d1c78721a746e99d0d3056f1e31371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 83, Last appended: 1.84, Last appended by leader: 84, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } }
I20240502 14:24:50.443261 14117 ts_tablet_manager.cc:1398] T d55e54737408455fadf1a1b4041edd0a P b0d1c78721a746e99d0d3056f1e31371: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:24:50.455930 12088 tablet_copy_service.cc:342] P 36e66c7b1a03458b89a91c56a57df72a: Request end of tablet copy session b0d1c78721a746e99d0d3056f1e31371-d55e54737408455fadf1a1b4041edd0a received from {username='slave'} at 127.5.107.71:39197
I20240502 14:24:50.456061 12088 tablet_copy_service.cc:433] P 36e66c7b1a03458b89a91c56a57df72a: ending tablet copy session b0d1c78721a746e99d0d3056f1e31371-d55e54737408455fadf1a1b4041edd0a on tablet d55e54737408455fadf1a1b4041edd0a with peer b0d1c78721a746e99d0d3056f1e31371
I20240502 14:24:50.462971 12196 consensus_queue.cc:237] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:24:50.465096 11936 raft_consensus.cc:1261] T bb1b842f76d74f6e996e48c245deda9b P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Refusing update from remote peer d9748c575342401bb80683c7e2c4e105: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch)
I20240502 14:24:50.465588 13854 consensus_queue.cc:1035] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20240502 14:24:50.468786 13854 raft_consensus.cc:2932] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 [term 1 LEADER]: Committing config change with OpId 1.106: config changed from index -1 to 106, VOTER 375c6d5bdf924866b69b0e6667732692 (127.5.107.70) added. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } } }
I20240502 14:24:50.473325 11936 raft_consensus.cc:2932] T bb1b842f76d74f6e996e48c245deda9b P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index -1 to 106, VOTER 375c6d5bdf924866b69b0e6667732692 (127.5.107.70) added. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } } }
I20240502 14:24:50.476500 13657 raft_consensus.cc:1203] T d55e54737408455fadf1a1b4041edd0a P b0d1c78721a746e99d0d3056f1e31371 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.70->[1.71-1.86] Dedup: 1.84->[1.85-1.86]
I20240502 14:24:50.481962 11787 catalog_manager.cc:5484] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 reported cstate change: config changed from index -1 to 106, VOTER 375c6d5bdf924866b69b0e6667732692 (127.5.107.70) added. New cstate: current_term: 1 leader_uuid: "d9748c575342401bb80683c7e2c4e105" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } } }
I20240502 14:24:50.526434 14157 ts_tablet_manager.cc:902] T 9069116816104c79819b25838836575b P 93c82e87d8af478292ffe1a3d3986049: Initiating tablet copy from peer d9748c575342401bb80683c7e2c4e105 (127.5.107.67:45923)
W20240502 14:24:50.548799 12135 consensus_peers.cc:485] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 -> Peer 375c6d5bdf924866b69b0e6667732692 (127.5.107.70:33951): Couldn't send request to peer 375c6d5bdf924866b69b0e6667732692. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb1b842f76d74f6e996e48c245deda9b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:24:50.554658 14157 tablet_copy_client.cc:308] T 9069116816104c79819b25838836575b P 93c82e87d8af478292ffe1a3d3986049: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.67:45923
I20240502 14:24:50.587690 12220 tablet_copy_service.cc:140] P d9748c575342401bb80683c7e2c4e105: Received BeginTabletCopySession request for tablet 9069116816104c79819b25838836575b from peer 93c82e87d8af478292ffe1a3d3986049 ({username='slave'} at 127.5.107.68:56023)
I20240502 14:24:50.587885 12220 tablet_copy_service.cc:161] P d9748c575342401bb80683c7e2c4e105: Beginning new tablet copy session on tablet 9069116816104c79819b25838836575b from peer 93c82e87d8af478292ffe1a3d3986049 at {username='slave'} at 127.5.107.68:56023: session id = 93c82e87d8af478292ffe1a3d3986049-9069116816104c79819b25838836575b
I20240502 14:24:50.590626 12220 tablet_copy_source_session.cc:205] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:24:50.594134 14157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9069116816104c79819b25838836575b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:50.617790 14157 tablet_copy_client.cc:783] T 9069116816104c79819b25838836575b P 93c82e87d8af478292ffe1a3d3986049: tablet copy: Starting download of 0 data blocks...
I20240502 14:24:50.618171 14157 tablet_copy_client.cc:647] T 9069116816104c79819b25838836575b P 93c82e87d8af478292ffe1a3d3986049: tablet copy: Starting download of 1 WAL segments...
I20240502 14:24:50.621205 14157 tablet_copy_client.cc:523] T 9069116816104c79819b25838836575b P 93c82e87d8af478292ffe1a3d3986049: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:24:50.624522 14157 tablet_bootstrap.cc:492] T 9069116816104c79819b25838836575b P 93c82e87d8af478292ffe1a3d3986049: Bootstrap starting.
I20240502 14:24:50.696874 11935 consensus_queue.cc:237] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 115, Committed index: 115, Last appended: 1.115, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } }
I20240502 14:24:50.718580 12197 raft_consensus.cc:1261] T 877f199681534650bc1d247cf7608f5f P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Refusing update from remote peer 0e41abe35d8f40a2a01953eea1c5d092: Log matching property violated. Preceding OpId in replica: term: 1 index: 115. Preceding OpId from leader: term: 1 index: 117. (index mismatch)
W20240502 14:24:50.733786 11868 consensus_peers.cc:485] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 -> Peer 723fb53f34c94c6aa5f7ee3a65711268 (127.5.107.69:33193): Couldn't send request to peer 723fb53f34c94c6aa5f7ee3a65711268. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 877f199681534650bc1d247cf7608f5f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:24:50.741294 14181 consensus_queue.cc:1035] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 116, Last known committed idx: 115, Time since last communication: 0.001s
I20240502 14:24:50.747882 13782 raft_consensus.cc:2932] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 LEADER]: Committing config change with OpId 1.116: config changed from index -1 to 116, VOTER 723fb53f34c94c6aa5f7ee3a65711268 (127.5.107.69) added. New config: { opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } }
I20240502 14:24:50.761149 12200 raft_consensus.cc:2932] T 877f199681534650bc1d247cf7608f5f P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Committing config change with OpId 1.116: config changed from index -1 to 116, VOTER 723fb53f34c94c6aa5f7ee3a65711268 (127.5.107.69) added. New config: { opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } }
I20240502 14:24:50.767066 11788 catalog_manager.cc:5484] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 reported cstate change: config changed from index -1 to 116, VOTER 723fb53f34c94c6aa5f7ee3a65711268 (127.5.107.69) added. New cstate: current_term: 1 leader_uuid: "0e41abe35d8f40a2a01953eea1c5d092" committed_config { opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } }
I20240502 14:24:50.819579 14157 tablet_bootstrap.cc:492] T 9069116816104c79819b25838836575b P 93c82e87d8af478292ffe1a3d3986049: Bootstrap replayed 1/1 log segments. Stats: ops{read=91 overwritten=0 applied=91 ignored=0} inserts{seen=776 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:24:50.820050 14157 tablet_bootstrap.cc:492] T 9069116816104c79819b25838836575b P 93c82e87d8af478292ffe1a3d3986049: Bootstrap complete.
I20240502 14:24:50.820391 14157 ts_tablet_manager.cc:1367] T 9069116816104c79819b25838836575b P 93c82e87d8af478292ffe1a3d3986049: Time spent bootstrapping tablet: real 0.196s user 0.138s sys 0.010s
I20240502 14:24:50.821506 14157 raft_consensus.cc:348] T 9069116816104c79819b25838836575b P 93c82e87d8af478292ffe1a3d3986049 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } }
I20240502 14:24:50.821835 14157 raft_consensus.cc:729] T 9069116816104c79819b25838836575b P 93c82e87d8af478292ffe1a3d3986049 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93c82e87d8af478292ffe1a3d3986049, State: Initialized, Role: FOLLOWER
I20240502 14:24:50.822077 14157 consensus_queue.cc:260] T 9069116816104c79819b25838836575b P 93c82e87d8af478292ffe1a3d3986049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 91, Last appended: 1.91, Last appended by leader: 91, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } }
I20240502 14:24:50.956594 14157 ts_tablet_manager.cc:1398] T 9069116816104c79819b25838836575b P 93c82e87d8af478292ffe1a3d3986049: Time spent starting tablet: real 0.136s user 0.055s sys 0.081s
I20240502 14:24:50.960992 12220 tablet_copy_service.cc:342] P d9748c575342401bb80683c7e2c4e105: Request end of tablet copy session 93c82e87d8af478292ffe1a3d3986049-9069116816104c79819b25838836575b received from {username='slave'} at 127.5.107.68:56023
I20240502 14:24:50.961151 12220 tablet_copy_service.cc:433] P d9748c575342401bb80683c7e2c4e105: ending tablet copy session 93c82e87d8af478292ffe1a3d3986049-9069116816104c79819b25838836575b on tablet 9069116816104c79819b25838836575b with peer 93c82e87d8af478292ffe1a3d3986049
I20240502 14:24:50.965988 14191 ts_tablet_manager.cc:902] T bb1b842f76d74f6e996e48c245deda9b P 375c6d5bdf924866b69b0e6667732692: Initiating tablet copy from peer d9748c575342401bb80683c7e2c4e105 (127.5.107.67:45923)
I20240502 14:24:50.978675 13251 raft_consensus.cc:1203] T 9069116816104c79819b25838836575b P 93c82e87d8af478292ffe1a3d3986049 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.84->[1.85-1.97] Dedup: 1.91->[1.92-1.97]
I20240502 14:24:51.111212 14191 tablet_copy_client.cc:308] T bb1b842f76d74f6e996e48c245deda9b P 375c6d5bdf924866b69b0e6667732692: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.67:45923
I20240502 14:24:51.249334 12220 tablet_copy_service.cc:140] P d9748c575342401bb80683c7e2c4e105: Received BeginTabletCopySession request for tablet bb1b842f76d74f6e996e48c245deda9b from peer 375c6d5bdf924866b69b0e6667732692 ({username='slave'} at 127.5.107.70:39029)
I20240502 14:24:51.249531 12220 tablet_copy_service.cc:161] P d9748c575342401bb80683c7e2c4e105: Beginning new tablet copy session on tablet bb1b842f76d74f6e996e48c245deda9b from peer 375c6d5bdf924866b69b0e6667732692 at {username='slave'} at 127.5.107.70:39029: session id = 375c6d5bdf924866b69b0e6667732692-bb1b842f76d74f6e996e48c245deda9b
I20240502 14:24:51.253197 12220 tablet_copy_source_session.cc:205] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:24:51.255472 14191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb1b842f76d74f6e996e48c245deda9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:51.261219 14191 tablet_copy_client.cc:783] T bb1b842f76d74f6e996e48c245deda9b P 375c6d5bdf924866b69b0e6667732692: tablet copy: Starting download of 0 data blocks...
I20240502 14:24:51.261456 14191 tablet_copy_client.cc:647] T bb1b842f76d74f6e996e48c245deda9b P 375c6d5bdf924866b69b0e6667732692: tablet copy: Starting download of 1 WAL segments...
I20240502 14:24:51.272750 14191 tablet_copy_client.cc:523] T bb1b842f76d74f6e996e48c245deda9b P 375c6d5bdf924866b69b0e6667732692: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:24:51.273797 14207 ts_tablet_manager.cc:902] T 877f199681534650bc1d247cf7608f5f P 723fb53f34c94c6aa5f7ee3a65711268: Initiating tablet copy from peer 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871)
I20240502 14:24:51.275962 14191 tablet_bootstrap.cc:492] T bb1b842f76d74f6e996e48c245deda9b P 375c6d5bdf924866b69b0e6667732692: Bootstrap starting.
I20240502 14:24:51.307628 14207 tablet_copy_client.cc:308] T 877f199681534650bc1d247cf7608f5f P 723fb53f34c94c6aa5f7ee3a65711268: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.65:36871
I20240502 14:24:51.337446 11956 tablet_copy_service.cc:140] P 0e41abe35d8f40a2a01953eea1c5d092: Received BeginTabletCopySession request for tablet 877f199681534650bc1d247cf7608f5f from peer 723fb53f34c94c6aa5f7ee3a65711268 ({username='slave'} at 127.5.107.69:56087)
I20240502 14:24:51.337847 11956 tablet_copy_service.cc:161] P 0e41abe35d8f40a2a01953eea1c5d092: Beginning new tablet copy session on tablet 877f199681534650bc1d247cf7608f5f from peer 723fb53f34c94c6aa5f7ee3a65711268 at {username='slave'} at 127.5.107.69:56087: session id = 723fb53f34c94c6aa5f7ee3a65711268-877f199681534650bc1d247cf7608f5f
I20240502 14:24:51.344157 11956 tablet_copy_source_session.cc:205] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:24:51.346611 14207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 877f199681534650bc1d247cf7608f5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:51.353575 14207 tablet_copy_client.cc:783] T 877f199681534650bc1d247cf7608f5f P 723fb53f34c94c6aa5f7ee3a65711268: tablet copy: Starting download of 0 data blocks...
I20240502 14:24:51.353891 14207 tablet_copy_client.cc:647] T 877f199681534650bc1d247cf7608f5f P 723fb53f34c94c6aa5f7ee3a65711268: tablet copy: Starting download of 1 WAL segments...
I20240502 14:24:51.356818 14207 tablet_copy_client.cc:523] T 877f199681534650bc1d247cf7608f5f P 723fb53f34c94c6aa5f7ee3a65711268: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:24:51.361644 14207 tablet_bootstrap.cc:492] T 877f199681534650bc1d247cf7608f5f P 723fb53f34c94c6aa5f7ee3a65711268: Bootstrap starting.
I20240502 14:24:51.492444 12068 consensus_queue.cc:237] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 1.127, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } }
W20240502 14:24:51.495100 12001 consensus_peers.cc:485] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a -> Peer b0d1c78721a746e99d0d3056f1e31371 (127.5.107.71:33799): Couldn't send request to peer b0d1c78721a746e99d0d3056f1e31371. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e7732ff4fa2744db94d0f8cb888de8d8. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:24:51.499537 12200 raft_consensus.cc:1261] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Refusing update from remote peer 36e66c7b1a03458b89a91c56a57df72a: Log matching property violated. Preceding OpId in replica: term: 1 index: 127. Preceding OpId from leader: term: 1 index: 128. (index mismatch)
I20240502 14:24:51.516968 13156 consensus_queue.cc:1035] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 128, Last known committed idx: 126, Time since last communication: 0.000s
I20240502 14:24:51.543507 13156 raft_consensus.cc:2932] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Committing config change with OpId 1.128: config changed from index -1 to 128, VOTER b0d1c78721a746e99d0d3056f1e31371 (127.5.107.71) added. New config: { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } }
I20240502 14:24:51.546898 12199 raft_consensus.cc:2932] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Committing config change with OpId 1.128: config changed from index -1 to 128, VOTER b0d1c78721a746e99d0d3056f1e31371 (127.5.107.71) added. New config: { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } }
I20240502 14:24:51.558706 11787 catalog_manager.cc:5484] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 reported cstate change: config changed from index -1 to 128, VOTER b0d1c78721a746e99d0d3056f1e31371 (127.5.107.71) added. New cstate: current_term: 1 leader_uuid: "36e66c7b1a03458b89a91c56a57df72a" committed_config { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } }
I20240502 14:24:51.763023 14191 tablet_bootstrap.cc:492] T bb1b842f76d74f6e996e48c245deda9b P 375c6d5bdf924866b69b0e6667732692: Bootstrap replayed 1/1 log segments. Stats: ops{read=116 overwritten=0 applied=115 ignored=0} inserts{seen=746 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:24:51.763475 14191 tablet_bootstrap.cc:492] T bb1b842f76d74f6e996e48c245deda9b P 375c6d5bdf924866b69b0e6667732692: Bootstrap complete.
I20240502 14:24:51.763787 14191 ts_tablet_manager.cc:1367] T bb1b842f76d74f6e996e48c245deda9b P 375c6d5bdf924866b69b0e6667732692: Time spent bootstrapping tablet: real 0.488s user 0.165s sys 0.007s
I20240502 14:24:51.764595 14191 raft_consensus.cc:348] T bb1b842f76d74f6e996e48c245deda9b P 375c6d5bdf924866b69b0e6667732692 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:24:51.772636 14191 raft_consensus.cc:729] T bb1b842f76d74f6e996e48c245deda9b P 375c6d5bdf924866b69b0e6667732692 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 375c6d5bdf924866b69b0e6667732692, State: Initialized, Role: FOLLOWER
I20240502 14:24:51.772864 14191 consensus_queue.cc:260] T bb1b842f76d74f6e996e48c245deda9b P 375c6d5bdf924866b69b0e6667732692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 115, Last appended: 1.116, Last appended by leader: 116, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:24:51.773375 14191 ts_tablet_manager.cc:1398] T bb1b842f76d74f6e996e48c245deda9b P 375c6d5bdf924866b69b0e6667732692: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240502 14:24:51.774061 12220 tablet_copy_service.cc:342] P d9748c575342401bb80683c7e2c4e105: Request end of tablet copy session 375c6d5bdf924866b69b0e6667732692-bb1b842f76d74f6e996e48c245deda9b received from {username='slave'} at 127.5.107.70:39029
I20240502 14:24:51.774201 12220 tablet_copy_service.cc:433] P d9748c575342401bb80683c7e2c4e105: ending tablet copy session 375c6d5bdf924866b69b0e6667732692-bb1b842f76d74f6e996e48c245deda9b on tablet bb1b842f76d74f6e996e48c245deda9b with peer 375c6d5bdf924866b69b0e6667732692
I20240502 14:24:51.810276 14207 tablet_bootstrap.cc:492] T 877f199681534650bc1d247cf7608f5f P 723fb53f34c94c6aa5f7ee3a65711268: Bootstrap replayed 1/1 log segments. Stats: ops{read=134 overwritten=0 applied=134 ignored=0} inserts{seen=819 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:24:51.810691 14207 tablet_bootstrap.cc:492] T 877f199681534650bc1d247cf7608f5f P 723fb53f34c94c6aa5f7ee3a65711268: Bootstrap complete.
I20240502 14:24:51.811018 14207 ts_tablet_manager.cc:1367] T 877f199681534650bc1d247cf7608f5f P 723fb53f34c94c6aa5f7ee3a65711268: Time spent bootstrapping tablet: real 0.449s user 0.192s sys 0.029s
I20240502 14:24:51.811744 14207 raft_consensus.cc:348] T 877f199681534650bc1d247cf7608f5f P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } }
I20240502 14:24:51.811985 14207 raft_consensus.cc:729] T 877f199681534650bc1d247cf7608f5f P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723fb53f34c94c6aa5f7ee3a65711268, State: Initialized, Role: FOLLOWER
I20240502 14:24:51.812203 14207 consensus_queue.cc:260] T 877f199681534650bc1d247cf7608f5f P 723fb53f34c94c6aa5f7ee3a65711268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 134, Last appended: 1.134, Last appended by leader: 134, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } }
I20240502 14:24:51.827836 14207 ts_tablet_manager.cc:1398] T 877f199681534650bc1d247cf7608f5f P 723fb53f34c94c6aa5f7ee3a65711268: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20240502 14:24:51.830113 11956 tablet_copy_service.cc:342] P 0e41abe35d8f40a2a01953eea1c5d092: Request end of tablet copy session 723fb53f34c94c6aa5f7ee3a65711268-877f199681534650bc1d247cf7608f5f received from {username='slave'} at 127.5.107.69:56087
I20240502 14:24:51.830289 11956 tablet_copy_service.cc:433] P 0e41abe35d8f40a2a01953eea1c5d092: ending tablet copy session 723fb53f34c94c6aa5f7ee3a65711268-877f199681534650bc1d247cf7608f5f on tablet 877f199681534650bc1d247cf7608f5f with peer 723fb53f34c94c6aa5f7ee3a65711268
I20240502 14:24:51.841989 13522 consensus_queue.cc:237] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } }
I20240502 14:24:51.848717 11935 raft_consensus.cc:1261] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [term 2 FOLLOWER]: Refusing update from remote peer 375c6d5bdf924866b69b0e6667732692: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:24:51.857883 14254 consensus_queue.cc:1035] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 14:24:51.864094 13456 consensus_peers.cc:485] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 -> Peer 93c82e87d8af478292ffe1a3d3986049 (127.5.107.68:37273): Couldn't send request to peer 93c82e87d8af478292ffe1a3d3986049. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f279550142c648d89b81feee9740f43e. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:24:51.870035 14231 raft_consensus.cc:2932] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 93c82e87d8af478292ffe1a3d3986049 (127.5.107.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } } }
I20240502 14:24:51.870707 11933 raft_consensus.cc:2932] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 93c82e87d8af478292ffe1a3d3986049 (127.5.107.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } } }
I20240502 14:24:51.887306 11788 catalog_manager.cc:5484] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 reported cstate change: config changed from index -1 to 3, VOTER 93c82e87d8af478292ffe1a3d3986049 (127.5.107.68) added. New cstate: current_term: 2 leader_uuid: "375c6d5bdf924866b69b0e6667732692" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } } }
I20240502 14:24:51.912618 14260 ts_tablet_manager.cc:902] T e7732ff4fa2744db94d0f8cb888de8d8 P b0d1c78721a746e99d0d3056f1e31371: Initiating tablet copy from peer 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445)
I20240502 14:24:51.988657 14260 tablet_copy_client.cc:308] T e7732ff4fa2744db94d0f8cb888de8d8 P b0d1c78721a746e99d0d3056f1e31371: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:35445
I20240502 14:24:51.989830 12088 tablet_copy_service.cc:140] P 36e66c7b1a03458b89a91c56a57df72a: Received BeginTabletCopySession request for tablet e7732ff4fa2744db94d0f8cb888de8d8 from peer b0d1c78721a746e99d0d3056f1e31371 ({username='slave'} at 127.5.107.71:39197)
I20240502 14:24:51.990023 12088 tablet_copy_service.cc:161] P 36e66c7b1a03458b89a91c56a57df72a: Beginning new tablet copy session on tablet e7732ff4fa2744db94d0f8cb888de8d8 from peer b0d1c78721a746e99d0d3056f1e31371 at {username='slave'} at 127.5.107.71:39197: session id = b0d1c78721a746e99d0d3056f1e31371-e7732ff4fa2744db94d0f8cb888de8d8
I20240502 14:24:51.992683 12088 tablet_copy_source_session.cc:205] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:24:51.994448 14260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7732ff4fa2744db94d0f8cb888de8d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:52.001348 14260 tablet_copy_client.cc:783] T e7732ff4fa2744db94d0f8cb888de8d8 P b0d1c78721a746e99d0d3056f1e31371: tablet copy: Starting download of 0 data blocks...
I20240502 14:24:52.001758 14260 tablet_copy_client.cc:647] T e7732ff4fa2744db94d0f8cb888de8d8 P b0d1c78721a746e99d0d3056f1e31371: tablet copy: Starting download of 1 WAL segments...
I20240502 14:24:52.004339 14260 tablet_copy_client.cc:523] T e7732ff4fa2744db94d0f8cb888de8d8 P b0d1c78721a746e99d0d3056f1e31371: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:24:52.007494 14260 tablet_bootstrap.cc:492] T e7732ff4fa2744db94d0f8cb888de8d8 P b0d1c78721a746e99d0d3056f1e31371: Bootstrap starting.
I20240502 14:24:52.087154 13522 raft_consensus.cc:1203] T bb1b842f76d74f6e996e48c245deda9b P 375c6d5bdf924866b69b0e6667732692 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.105->[1.106-1.135] Dedup: 1.116->[1.117-1.135]
W20240502 14:24:52.191720 14274 log.cc:924] Time spent T bb1b842f76d74f6e996e48c245deda9b P 375c6d5bdf924866b69b0e6667732692: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20240502 14:24:52.215108 13387 raft_consensus.cc:1203] T 877f199681534650bc1d247cf7608f5f P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.115->[1.116-1.157] Dedup: 1.134->[1.135-1.157]
I20240502 14:24:52.403016 14260 tablet_bootstrap.cc:492] T e7732ff4fa2744db94d0f8cb888de8d8 P b0d1c78721a746e99d0d3056f1e31371: Bootstrap replayed 1/1 log segments. Stats: ops{read=133 overwritten=0 applied=132 ignored=0} inserts{seen=763 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:24:52.403540 14260 tablet_bootstrap.cc:492] T e7732ff4fa2744db94d0f8cb888de8d8 P b0d1c78721a746e99d0d3056f1e31371: Bootstrap complete.
I20240502 14:24:52.403895 14260 ts_tablet_manager.cc:1367] T e7732ff4fa2744db94d0f8cb888de8d8 P b0d1c78721a746e99d0d3056f1e31371: Time spent bootstrapping tablet: real 0.396s user 0.185s sys 0.007s
I20240502 14:24:52.404779 14260 raft_consensus.cc:348] T e7732ff4fa2744db94d0f8cb888de8d8 P b0d1c78721a746e99d0d3056f1e31371 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } }
I20240502 14:24:52.405479 14260 raft_consensus.cc:729] T e7732ff4fa2744db94d0f8cb888de8d8 P b0d1c78721a746e99d0d3056f1e31371 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0d1c78721a746e99d0d3056f1e31371, State: Initialized, Role: FOLLOWER
I20240502 14:24:52.405843 14260 consensus_queue.cc:260] T e7732ff4fa2744db94d0f8cb888de8d8 P b0d1c78721a746e99d0d3056f1e31371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 132, Last appended: 1.133, Last appended by leader: 133, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } }
I20240502 14:24:52.431721 14260 ts_tablet_manager.cc:1398] T e7732ff4fa2744db94d0f8cb888de8d8 P b0d1c78721a746e99d0d3056f1e31371: Time spent starting tablet: real 0.028s user 0.015s sys 0.011s
I20240502 14:24:52.432524 12088 tablet_copy_service.cc:342] P 36e66c7b1a03458b89a91c56a57df72a: Request end of tablet copy session b0d1c78721a746e99d0d3056f1e31371-e7732ff4fa2744db94d0f8cb888de8d8 received from {username='slave'} at 127.5.107.71:39197
I20240502 14:24:52.432675 12088 tablet_copy_service.cc:433] P 36e66c7b1a03458b89a91c56a57df72a: ending tablet copy session b0d1c78721a746e99d0d3056f1e31371-e7732ff4fa2744db94d0f8cb888de8d8 on tablet e7732ff4fa2744db94d0f8cb888de8d8 with peer b0d1c78721a746e99d0d3056f1e31371
W20240502 14:24:52.525358 11999 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.69:33193, user_credentials={real_user=slave}} blocked reactor thread for 133746us
I20240502 14:24:52.537722 13657 raft_consensus.cc:1203] T e7732ff4fa2744db94d0f8cb888de8d8 P b0d1c78721a746e99d0d3056f1e31371 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.127->[1.128-1.134] Dedup: 1.133->[1.134-1.134]
I20240502 14:24:52.635357 12068 consensus_queue.cc:237] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 154, Committed index: 154, Last appended: 1.155, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
W20240502 14:24:52.638252 12002 consensus_peers.cc:485] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a -> Peer 375c6d5bdf924866b69b0e6667732692 (127.5.107.70:33951): Couldn't send request to peer 375c6d5bdf924866b69b0e6667732692. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f0a86c3cb6c48258f4d2f916b192c9c. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:24:52.668658 11936 raft_consensus.cc:1261] T 2f0a86c3cb6c48258f4d2f916b192c9c P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Refusing update from remote peer 36e66c7b1a03458b89a91c56a57df72a: Log matching property violated. Preceding OpId in replica: term: 1 index: 155. Preceding OpId from leader: term: 1 index: 156. (index mismatch)
I20240502 14:24:52.676698 14293 consensus_queue.cc:1035] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 156, Last known committed idx: 154, Time since last communication: 0.000s
I20240502 14:24:52.679667 14301 ts_tablet_manager.cc:902] T f279550142c648d89b81feee9740f43e P 93c82e87d8af478292ffe1a3d3986049: Initiating tablet copy from peer 375c6d5bdf924866b69b0e6667732692 (127.5.107.70:33951)
I20240502 14:24:52.690971 13156 raft_consensus.cc:2932] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Committing config change with OpId 1.156: config changed from index -1 to 156, VOTER 375c6d5bdf924866b69b0e6667732692 (127.5.107.70) added. New config: { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } } }
I20240502 14:24:52.698812 11935 raft_consensus.cc:2932] T 2f0a86c3cb6c48258f4d2f916b192c9c P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Committing config change with OpId 1.156: config changed from index -1 to 156, VOTER 375c6d5bdf924866b69b0e6667732692 (127.5.107.70) added. New config: { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } } }
I20240502 14:24:52.705008 14301 tablet_copy_client.cc:308] T f279550142c648d89b81feee9740f43e P 93c82e87d8af478292ffe1a3d3986049: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.70:33951
I20240502 14:24:52.705549 11788 catalog_manager.cc:5484] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a reported cstate change: config changed from index -1 to 156, VOTER 375c6d5bdf924866b69b0e6667732692 (127.5.107.70) added. New cstate: current_term: 1 leader_uuid: "36e66c7b1a03458b89a91c56a57df72a" committed_config { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } } }
I20240502 14:24:52.864671 13542 tablet_copy_service.cc:140] P 375c6d5bdf924866b69b0e6667732692: Received BeginTabletCopySession request for tablet f279550142c648d89b81feee9740f43e from peer 93c82e87d8af478292ffe1a3d3986049 ({username='slave'} at 127.5.107.68:34885)
I20240502 14:24:52.864959 13542 tablet_copy_service.cc:161] P 375c6d5bdf924866b69b0e6667732692: Beginning new tablet copy session on tablet f279550142c648d89b81feee9740f43e from peer 93c82e87d8af478292ffe1a3d3986049 at {username='slave'} at 127.5.107.68:34885: session id = 93c82e87d8af478292ffe1a3d3986049-f279550142c648d89b81feee9740f43e
I20240502 14:24:52.869184 13542 tablet_copy_source_session.cc:205] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:24:52.873073 14301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f279550142c648d89b81feee9740f43e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:52.880147 14301 tablet_copy_client.cc:783] T f279550142c648d89b81feee9740f43e P 93c82e87d8af478292ffe1a3d3986049: tablet copy: Starting download of 0 data blocks...
I20240502 14:24:52.880609 14301 tablet_copy_client.cc:647] T f279550142c648d89b81feee9740f43e P 93c82e87d8af478292ffe1a3d3986049: tablet copy: Starting download of 1 WAL segments...
I20240502 14:24:53.021294 14301 tablet_copy_client.cc:523] T f279550142c648d89b81feee9740f43e P 93c82e87d8af478292ffe1a3d3986049: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:24:53.079668 14301 tablet_bootstrap.cc:492] T f279550142c648d89b81feee9740f43e P 93c82e87d8af478292ffe1a3d3986049: Bootstrap starting.
I20240502 14:24:53.139753 14328 ts_tablet_manager.cc:902] T 2f0a86c3cb6c48258f4d2f916b192c9c P 375c6d5bdf924866b69b0e6667732692: Initiating tablet copy from peer 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445)
I20240502 14:24:53.142995 14328 tablet_copy_client.cc:308] T 2f0a86c3cb6c48258f4d2f916b192c9c P 375c6d5bdf924866b69b0e6667732692: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:35445
I20240502 14:24:53.166452 12088 tablet_copy_service.cc:140] P 36e66c7b1a03458b89a91c56a57df72a: Received BeginTabletCopySession request for tablet 2f0a86c3cb6c48258f4d2f916b192c9c from peer 375c6d5bdf924866b69b0e6667732692 ({username='slave'} at 127.5.107.70:60345)
I20240502 14:24:53.166612 12088 tablet_copy_service.cc:161] P 36e66c7b1a03458b89a91c56a57df72a: Beginning new tablet copy session on tablet 2f0a86c3cb6c48258f4d2f916b192c9c from peer 375c6d5bdf924866b69b0e6667732692 at {username='slave'} at 127.5.107.70:60345: session id = 375c6d5bdf924866b69b0e6667732692-2f0a86c3cb6c48258f4d2f916b192c9c
I20240502 14:24:53.168109 14301 tablet_bootstrap.cc:492] T f279550142c648d89b81feee9740f43e P 93c82e87d8af478292ffe1a3d3986049: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:24:53.168569 14301 tablet_bootstrap.cc:492] T f279550142c648d89b81feee9740f43e P 93c82e87d8af478292ffe1a3d3986049: Bootstrap complete.
I20240502 14:24:53.168845 14301 ts_tablet_manager.cc:1367] T f279550142c648d89b81feee9740f43e P 93c82e87d8af478292ffe1a3d3986049: Time spent bootstrapping tablet: real 0.089s user 0.003s sys 0.003s
I20240502 14:24:53.169730 14301 raft_consensus.cc:348] T f279550142c648d89b81feee9740f43e P 93c82e87d8af478292ffe1a3d3986049 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } }
I20240502 14:24:53.169941 14301 raft_consensus.cc:729] T f279550142c648d89b81feee9740f43e P 93c82e87d8af478292ffe1a3d3986049 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93c82e87d8af478292ffe1a3d3986049, State: Initialized, Role: FOLLOWER
I20240502 14:24:53.170264 14301 consensus_queue.cc:260] T f279550142c648d89b81feee9740f43e P 93c82e87d8af478292ffe1a3d3986049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } }
I20240502 14:24:53.172737 14301 ts_tablet_manager.cc:1398] T f279550142c648d89b81feee9740f43e P 93c82e87d8af478292ffe1a3d3986049: Time spent starting tablet: real 0.004s user 0.006s sys 0.001s
I20240502 14:24:53.175168 13542 tablet_copy_service.cc:342] P 375c6d5bdf924866b69b0e6667732692: Request end of tablet copy session 93c82e87d8af478292ffe1a3d3986049-f279550142c648d89b81feee9740f43e received from {username='slave'} at 127.5.107.68:34885
I20240502 14:24:53.175305 13542 tablet_copy_service.cc:433] P 375c6d5bdf924866b69b0e6667732692: ending tablet copy session 93c82e87d8af478292ffe1a3d3986049-f279550142c648d89b81feee9740f43e on tablet f279550142c648d89b81feee9740f43e with peer 93c82e87d8af478292ffe1a3d3986049
I20240502 14:24:53.169210 12088 tablet_copy_source_session.cc:205] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:24:53.182507 14328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f0a86c3cb6c48258f4d2f916b192c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:53.233414 14328 tablet_copy_client.cc:783] T 2f0a86c3cb6c48258f4d2f916b192c9c P 375c6d5bdf924866b69b0e6667732692: tablet copy: Starting download of 0 data blocks...
I20240502 14:24:53.233665 14328 tablet_copy_client.cc:647] T 2f0a86c3cb6c48258f4d2f916b192c9c P 375c6d5bdf924866b69b0e6667732692: tablet copy: Starting download of 1 WAL segments...
I20240502 14:24:53.249802 14328 tablet_copy_client.cc:523] T 2f0a86c3cb6c48258f4d2f916b192c9c P 375c6d5bdf924866b69b0e6667732692: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:24:53.277663 14328 tablet_bootstrap.cc:492] T 2f0a86c3cb6c48258f4d2f916b192c9c P 375c6d5bdf924866b69b0e6667732692: Bootstrap starting.
I20240502 14:24:53.475288 13250 raft_consensus.cc:1203] T f279550142c648d89b81feee9740f43e P 93c82e87d8af478292ffe1a3d3986049 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240502 14:24:53.677093 14328 tablet_bootstrap.cc:492] T 2f0a86c3cb6c48258f4d2f916b192c9c P 375c6d5bdf924866b69b0e6667732692: Bootstrap replayed 1/1 log segments. Stats: ops{read=164 overwritten=0 applied=164 ignored=0} inserts{seen=836 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:24:53.677788 14328 tablet_bootstrap.cc:492] T 2f0a86c3cb6c48258f4d2f916b192c9c P 375c6d5bdf924866b69b0e6667732692: Bootstrap complete.
I20240502 14:24:53.678146 14328 ts_tablet_manager.cc:1367] T 2f0a86c3cb6c48258f4d2f916b192c9c P 375c6d5bdf924866b69b0e6667732692: Time spent bootstrapping tablet: real 0.401s user 0.228s sys 0.012s
I20240502 14:24:53.678946 14328 raft_consensus.cc:348] T 2f0a86c3cb6c48258f4d2f916b192c9c P 375c6d5bdf924866b69b0e6667732692 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:24:53.679280 14328 raft_consensus.cc:729] T 2f0a86c3cb6c48258f4d2f916b192c9c P 375c6d5bdf924866b69b0e6667732692 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 375c6d5bdf924866b69b0e6667732692, State: Initialized, Role: FOLLOWER
I20240502 14:24:53.679600 14328 consensus_queue.cc:260] T 2f0a86c3cb6c48258f4d2f916b192c9c P 375c6d5bdf924866b69b0e6667732692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 164, Last appended: 1.164, Last appended by leader: 164, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:24:53.680163 14328 ts_tablet_manager.cc:1398] T 2f0a86c3cb6c48258f4d2f916b192c9c P 375c6d5bdf924866b69b0e6667732692: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:24:53.684392 12088 tablet_copy_service.cc:342] P 36e66c7b1a03458b89a91c56a57df72a: Request end of tablet copy session 375c6d5bdf924866b69b0e6667732692-2f0a86c3cb6c48258f4d2f916b192c9c received from {username='slave'} at 127.5.107.70:60345
I20240502 14:24:53.684543 12088 tablet_copy_service.cc:433] P 36e66c7b1a03458b89a91c56a57df72a: ending tablet copy session 375c6d5bdf924866b69b0e6667732692-2f0a86c3cb6c48258f4d2f916b192c9c on tablet 2f0a86c3cb6c48258f4d2f916b192c9c with peer 375c6d5bdf924866b69b0e6667732692
I20240502 14:24:53.707705 13521 raft_consensus.cc:1203] T 2f0a86c3cb6c48258f4d2f916b192c9c P 375c6d5bdf924866b69b0e6667732692 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.155->[1.156-1.171] Dedup: 1.164->[1.165-1.171]
I20240502 14:24:53.846434 12068 consensus_queue.cc:237] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 147, Committed index: 147, Last appended: 1.147, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 148 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } }
I20240502 14:24:53.849824 12200 raft_consensus.cc:1261] T c3811b0d595f4292b9d96454deba8156 P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Refusing update from remote peer 36e66c7b1a03458b89a91c56a57df72a: Log matching property violated. Preceding OpId in replica: term: 1 index: 147. Preceding OpId from leader: term: 1 index: 148. (index mismatch)
W20240502 14:24:53.850371 11999 consensus_peers.cc:485] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a -> Peer 723fb53f34c94c6aa5f7ee3a65711268 (127.5.107.69:33193): Couldn't send request to peer 723fb53f34c94c6aa5f7ee3a65711268. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3811b0d595f4292b9d96454deba8156. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:24:53.853154 13802 consensus_queue.cc:1035] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 148, Last known committed idx: 147, Time since last communication: 0.000s
I20240502 14:24:53.857656 13781 raft_consensus.cc:2932] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Committing config change with OpId 1.148: config changed from index -1 to 148, VOTER 723fb53f34c94c6aa5f7ee3a65711268 (127.5.107.69) added. New config: { opid_index: 148 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } }
I20240502 14:24:53.863159 11787 catalog_manager.cc:5484] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a reported cstate change: config changed from index -1 to 148, VOTER 723fb53f34c94c6aa5f7ee3a65711268 (127.5.107.69) added. New cstate: current_term: 1 leader_uuid: "36e66c7b1a03458b89a91c56a57df72a" committed_config { opid_index: 148 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } }
I20240502 14:24:53.872895 12200 raft_consensus.cc:2932] T c3811b0d595f4292b9d96454deba8156 P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Committing config change with OpId 1.148: config changed from index -1 to 148, VOTER 723fb53f34c94c6aa5f7ee3a65711268 (127.5.107.69) added. New config: { opid_index: 148 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } }
W20240502 14:24:53.972888 11999 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.67:45923, user_credentials={real_user=slave}} blocked reactor thread for 48069us
I20240502 14:24:54.362800 12068 consensus_queue.cc:237] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 139, Committed index: 139, Last appended: 1.139, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } }
I20240502 14:24:54.364938 12195 raft_consensus.cc:1261] T e05a3a4fb76d4c46908945ad0d838509 P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Refusing update from remote peer 36e66c7b1a03458b89a91c56a57df72a: Log matching property violated. Preceding OpId in replica: term: 1 index: 139. Preceding OpId from leader: term: 1 index: 140. (index mismatch)
W20240502 14:24:54.365881 12001 consensus_peers.cc:485] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a -> Peer b0d1c78721a746e99d0d3056f1e31371 (127.5.107.71:33799): Couldn't send request to peer b0d1c78721a746e99d0d3056f1e31371. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e05a3a4fb76d4c46908945ad0d838509. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:24:54.373436 13781 consensus_queue.cc:1035] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 140, Last known committed idx: 139, Time since last communication: 0.000s
I20240502 14:24:54.384469 14292 raft_consensus.cc:2932] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Committing config change with OpId 1.140: config changed from index -1 to 140, VOTER b0d1c78721a746e99d0d3056f1e31371 (127.5.107.71) added. New config: { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } }
I20240502 14:24:54.394456 12199 raft_consensus.cc:2932] T e05a3a4fb76d4c46908945ad0d838509 P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Committing config change with OpId 1.140: config changed from index -1 to 140, VOTER b0d1c78721a746e99d0d3056f1e31371 (127.5.107.71) added. New config: { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } }
I20240502 14:24:54.399784 11788 catalog_manager.cc:5484] T e05a3a4fb76d4c46908945ad0d838509 P d9748c575342401bb80683c7e2c4e105 reported cstate change: config changed from index -1 to 140, VOTER b0d1c78721a746e99d0d3056f1e31371 (127.5.107.71) added. New cstate: current_term: 1 leader_uuid: "36e66c7b1a03458b89a91c56a57df72a" committed_config { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } }
I20240502 14:24:54.444378 14386 ts_tablet_manager.cc:902] T c3811b0d595f4292b9d96454deba8156 P 723fb53f34c94c6aa5f7ee3a65711268: Initiating tablet copy from peer 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445)
I20240502 14:24:54.497136 14386 tablet_copy_client.cc:308] T c3811b0d595f4292b9d96454deba8156 P 723fb53f34c94c6aa5f7ee3a65711268: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:35445
I20240502 14:24:54.499127 12088 tablet_copy_service.cc:140] P 36e66c7b1a03458b89a91c56a57df72a: Received BeginTabletCopySession request for tablet c3811b0d595f4292b9d96454deba8156 from peer 723fb53f34c94c6aa5f7ee3a65711268 ({username='slave'} at 127.5.107.69:39951)
I20240502 14:24:54.499354 12088 tablet_copy_service.cc:161] P 36e66c7b1a03458b89a91c56a57df72a: Beginning new tablet copy session on tablet c3811b0d595f4292b9d96454deba8156 from peer 723fb53f34c94c6aa5f7ee3a65711268 at {username='slave'} at 127.5.107.69:39951: session id = 723fb53f34c94c6aa5f7ee3a65711268-c3811b0d595f4292b9d96454deba8156
I20240502 14:24:54.503144 12088 tablet_copy_source_session.cc:205] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:24:54.505748 14386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3811b0d595f4292b9d96454deba8156. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:54.512486 14386 tablet_copy_client.cc:783] T c3811b0d595f4292b9d96454deba8156 P 723fb53f34c94c6aa5f7ee3a65711268: tablet copy: Starting download of 0 data blocks...
I20240502 14:24:54.512774 14386 tablet_copy_client.cc:647] T c3811b0d595f4292b9d96454deba8156 P 723fb53f34c94c6aa5f7ee3a65711268: tablet copy: Starting download of 1 WAL segments...
I20240502 14:24:54.531730 14386 tablet_copy_client.cc:523] T c3811b0d595f4292b9d96454deba8156 P 723fb53f34c94c6aa5f7ee3a65711268: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:24:54.549420 14386 tablet_bootstrap.cc:492] T c3811b0d595f4292b9d96454deba8156 P 723fb53f34c94c6aa5f7ee3a65711268: Bootstrap starting.
I20240502 14:24:54.777199 14403 ts_tablet_manager.cc:902] T e05a3a4fb76d4c46908945ad0d838509 P b0d1c78721a746e99d0d3056f1e31371: Initiating tablet copy from peer 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445)
I20240502 14:24:54.817771 14403 tablet_copy_client.cc:308] T e05a3a4fb76d4c46908945ad0d838509 P b0d1c78721a746e99d0d3056f1e31371: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:35445
I20240502 14:24:54.818579 12088 tablet_copy_service.cc:140] P 36e66c7b1a03458b89a91c56a57df72a: Received BeginTabletCopySession request for tablet e05a3a4fb76d4c46908945ad0d838509 from peer b0d1c78721a746e99d0d3056f1e31371 ({username='slave'} at 127.5.107.71:39197)
I20240502 14:24:54.818748 12088 tablet_copy_service.cc:161] P 36e66c7b1a03458b89a91c56a57df72a: Beginning new tablet copy session on tablet e05a3a4fb76d4c46908945ad0d838509 from peer b0d1c78721a746e99d0d3056f1e31371 at {username='slave'} at 127.5.107.71:39197: session id = b0d1c78721a746e99d0d3056f1e31371-e05a3a4fb76d4c46908945ad0d838509
I20240502 14:24:54.823650 12088 tablet_copy_source_session.cc:205] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:24:54.825485 14403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e05a3a4fb76d4c46908945ad0d838509. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:54.834323 14403 tablet_copy_client.cc:783] T e05a3a4fb76d4c46908945ad0d838509 P b0d1c78721a746e99d0d3056f1e31371: tablet copy: Starting download of 0 data blocks...
I20240502 14:24:54.834928 14403 tablet_copy_client.cc:647] T e05a3a4fb76d4c46908945ad0d838509 P b0d1c78721a746e99d0d3056f1e31371: tablet copy: Starting download of 1 WAL segments...
I20240502 14:24:54.846547 14403 tablet_copy_client.cc:523] T e05a3a4fb76d4c46908945ad0d838509 P b0d1c78721a746e99d0d3056f1e31371: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:24:54.862314 14403 tablet_bootstrap.cc:492] T e05a3a4fb76d4c46908945ad0d838509 P b0d1c78721a746e99d0d3056f1e31371: Bootstrap starting.
I20240502 14:24:55.082118 14386 tablet_bootstrap.cc:492] T c3811b0d595f4292b9d96454deba8156 P 723fb53f34c94c6aa5f7ee3a65711268: Bootstrap replayed 1/1 log segments. Stats: ops{read=157 overwritten=0 applied=156 ignored=0} inserts{seen=841 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:24:55.083650 14386 tablet_bootstrap.cc:492] T c3811b0d595f4292b9d96454deba8156 P 723fb53f34c94c6aa5f7ee3a65711268: Bootstrap complete.
I20240502 14:24:55.084311 14386 ts_tablet_manager.cc:1367] T c3811b0d595f4292b9d96454deba8156 P 723fb53f34c94c6aa5f7ee3a65711268: Time spent bootstrapping tablet: real 0.535s user 0.210s sys 0.060s
I20240502 14:24:55.085464 14386 raft_consensus.cc:348] T c3811b0d595f4292b9d96454deba8156 P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 148 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } }
I20240502 14:24:55.086743 14386 raft_consensus.cc:729] T c3811b0d595f4292b9d96454deba8156 P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723fb53f34c94c6aa5f7ee3a65711268, State: Initialized, Role: FOLLOWER
I20240502 14:24:55.087009 14386 consensus_queue.cc:260] T c3811b0d595f4292b9d96454deba8156 P 723fb53f34c94c6aa5f7ee3a65711268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 156, Last appended: 1.157, Last appended by leader: 157, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 148 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } }
I20240502 14:24:55.089105 14386 ts_tablet_manager.cc:1398] T c3811b0d595f4292b9d96454deba8156 P 723fb53f34c94c6aa5f7ee3a65711268: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:24:55.089962 12088 tablet_copy_service.cc:342] P 36e66c7b1a03458b89a91c56a57df72a: Request end of tablet copy session 723fb53f34c94c6aa5f7ee3a65711268-c3811b0d595f4292b9d96454deba8156 received from {username='slave'} at 127.5.107.69:39951
I20240502 14:24:55.090102 12088 tablet_copy_service.cc:433] P 36e66c7b1a03458b89a91c56a57df72a: ending tablet copy session 723fb53f34c94c6aa5f7ee3a65711268-c3811b0d595f4292b9d96454deba8156 on tablet c3811b0d595f4292b9d96454deba8156 with peer 723fb53f34c94c6aa5f7ee3a65711268
I20240502 14:24:55.179658 12068 consensus_queue.cc:237] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 148, Committed index: 148, Last appended: 1.149, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } }
W20240502 14:24:55.182431 11999 consensus_peers.cc:485] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a -> Peer 723fb53f34c94c6aa5f7ee3a65711268 (127.5.107.69:33193): Couldn't send request to peer 723fb53f34c94c6aa5f7ee3a65711268. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1427194e32d545f785df3e45137aa42e. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:24:55.202740 12196 raft_consensus.cc:1261] T 1427194e32d545f785df3e45137aa42e P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Refusing update from remote peer 36e66c7b1a03458b89a91c56a57df72a: Log matching property violated. Preceding OpId in replica: term: 1 index: 149. Preceding OpId from leader: term: 1 index: 150. (index mismatch)
I20240502 14:24:55.219568 14294 consensus_queue.cc:1035] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 150, Last known committed idx: 148, Time since last communication: 0.000s
I20240502 14:24:55.240301 14403 tablet_bootstrap.cc:492] T e05a3a4fb76d4c46908945ad0d838509 P b0d1c78721a746e99d0d3056f1e31371: Bootstrap replayed 1/1 log segments. Stats: ops{read=146 overwritten=0 applied=146 ignored=0} inserts{seen=777 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:24:55.241209 14403 tablet_bootstrap.cc:492] T e05a3a4fb76d4c46908945ad0d838509 P b0d1c78721a746e99d0d3056f1e31371: Bootstrap complete.
I20240502 14:24:55.241583 14403 ts_tablet_manager.cc:1367] T e05a3a4fb76d4c46908945ad0d838509 P b0d1c78721a746e99d0d3056f1e31371: Time spent bootstrapping tablet: real 0.380s user 0.195s sys 0.025s
I20240502 14:24:55.242527 14403 raft_consensus.cc:348] T e05a3a4fb76d4c46908945ad0d838509 P b0d1c78721a746e99d0d3056f1e31371 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } }
I20240502 14:24:55.243237 14403 raft_consensus.cc:729] T e05a3a4fb76d4c46908945ad0d838509 P b0d1c78721a746e99d0d3056f1e31371 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0d1c78721a746e99d0d3056f1e31371, State: Initialized, Role: FOLLOWER
I20240502 14:24:55.244192 14403 consensus_queue.cc:260] T e05a3a4fb76d4c46908945ad0d838509 P b0d1c78721a746e99d0d3056f1e31371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 146, Last appended: 1.146, Last appended by leader: 146, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } }
I20240502 14:24:55.246150 13858 raft_consensus.cc:2932] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Committing config change with OpId 1.150: config changed from index -1 to 150, VOTER 723fb53f34c94c6aa5f7ee3a65711268 (127.5.107.69) added. New config: { opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } }
I20240502 14:24:55.251545 14403 ts_tablet_manager.cc:1398] T e05a3a4fb76d4c46908945ad0d838509 P b0d1c78721a746e99d0d3056f1e31371: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20240502 14:24:55.251801 13655 raft_consensus.cc:1203] T e05a3a4fb76d4c46908945ad0d838509 P b0d1c78721a746e99d0d3056f1e31371 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.139->[1.140-1.148] Dedup: 1.146->[1.147-1.148]
I20240502 14:24:55.252640 12088 tablet_copy_service.cc:342] P 36e66c7b1a03458b89a91c56a57df72a: Request end of tablet copy session b0d1c78721a746e99d0d3056f1e31371-e05a3a4fb76d4c46908945ad0d838509 received from {username='slave'} at 127.5.107.71:39197
I20240502 14:24:55.252791 12088 tablet_copy_service.cc:433] P 36e66c7b1a03458b89a91c56a57df72a: ending tablet copy session b0d1c78721a746e99d0d3056f1e31371-e05a3a4fb76d4c46908945ad0d838509 on tablet e05a3a4fb76d4c46908945ad0d838509 with peer b0d1c78721a746e99d0d3056f1e31371
I20240502 14:24:55.261380 11788 catalog_manager.cc:5484] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a reported cstate change: config changed from index -1 to 150, VOTER 723fb53f34c94c6aa5f7ee3a65711268 (127.5.107.69) added. New cstate: current_term: 1 leader_uuid: "36e66c7b1a03458b89a91c56a57df72a" committed_config { opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } }
I20240502 14:24:55.291476 12197 raft_consensus.cc:2932] T 1427194e32d545f785df3e45137aa42e P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Committing config change with OpId 1.150: config changed from index -1 to 150, VOTER 723fb53f34c94c6aa5f7ee3a65711268 (127.5.107.69) added. New config: { opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } }
I20240502 14:24:55.337869 13387 raft_consensus.cc:1203] T c3811b0d595f4292b9d96454deba8156 P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.147->[1.148-1.164] Dedup: 1.157->[1.158-1.164]
W20240502 14:24:55.517287 12133 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.65:36871, user_credentials={real_user=slave}} blocked reactor thread for 91460.7us
W20240502 14:24:55.650877 11868 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.67:45923, user_credentials={real_user=slave}} blocked reactor thread for 87015.5us
I20240502 14:24:55.782235 14460 ts_tablet_manager.cc:902] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268: Initiating tablet copy from peer 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445)
I20240502 14:24:55.900189 14460 tablet_copy_client.cc:308] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:35445
I20240502 14:24:55.901262 12088 tablet_copy_service.cc:140] P 36e66c7b1a03458b89a91c56a57df72a: Received BeginTabletCopySession request for tablet 1427194e32d545f785df3e45137aa42e from peer 723fb53f34c94c6aa5f7ee3a65711268 ({username='slave'} at 127.5.107.69:39951)
I20240502 14:24:55.901468 12088 tablet_copy_service.cc:161] P 36e66c7b1a03458b89a91c56a57df72a: Beginning new tablet copy session on tablet 1427194e32d545f785df3e45137aa42e from peer 723fb53f34c94c6aa5f7ee3a65711268 at {username='slave'} at 127.5.107.69:39951: session id = 723fb53f34c94c6aa5f7ee3a65711268-1427194e32d545f785df3e45137aa42e
I20240502 14:24:55.903350 12088 tablet_copy_source_session.cc:205] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:24:55.905349 14460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1427194e32d545f785df3e45137aa42e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:24:55.921373 14460 tablet_copy_client.cc:783] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268: tablet copy: Starting download of 0 data blocks...
I20240502 14:24:55.921723 14460 tablet_copy_client.cc:647] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268: tablet copy: Starting download of 1 WAL segments...
I20240502 14:24:55.949893 14460 tablet_copy_client.cc:523] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:24:55.954133 14460 tablet_bootstrap.cc:492] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268: Bootstrap starting.
W20240502 14:24:56.044246 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.65:36871, user_credentials={real_user=slave}} blocked reactor thread for 128548us
I20240502 14:24:56.672087 14460 tablet_bootstrap.cc:492] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268: Bootstrap replayed 1/1 log segments. Stats: ops{read=160 overwritten=0 applied=160 ignored=0} inserts{seen=791 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:24:56.675371 14460 tablet_bootstrap.cc:492] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268: Bootstrap complete.
I20240502 14:24:56.682757 14460 ts_tablet_manager.cc:1367] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268: Time spent bootstrapping tablet: real 0.729s user 0.246s sys 0.021s
I20240502 14:24:56.698995 14460 raft_consensus.cc:348] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } }
I20240502 14:24:56.699317 14460 raft_consensus.cc:729] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723fb53f34c94c6aa5f7ee3a65711268, State: Initialized, Role: FOLLOWER
I20240502 14:24:56.701342 14460 consensus_queue.cc:260] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 160, Last appended: 1.160, Last appended by leader: 160, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } }
I20240502 14:24:56.717435 14460 ts_tablet_manager.cc:1398] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268: Time spent starting tablet: real 0.034s user 0.002s sys 0.001s
I20240502 14:24:56.718204 12088 tablet_copy_service.cc:342] P 36e66c7b1a03458b89a91c56a57df72a: Request end of tablet copy session 723fb53f34c94c6aa5f7ee3a65711268-1427194e32d545f785df3e45137aa42e received from {username='slave'} at 127.5.107.69:39951
I20240502 14:24:56.718343 12088 tablet_copy_service.cc:433] P 36e66c7b1a03458b89a91c56a57df72a: ending tablet copy session 723fb53f34c94c6aa5f7ee3a65711268-1427194e32d545f785df3e45137aa42e on tablet 1427194e32d545f785df3e45137aa42e with peer 723fb53f34c94c6aa5f7ee3a65711268
I20240502 14:24:56.825691 13386 raft_consensus.cc:1203] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.149->[1.150-1.169] Dedup: 1.160->[1.161-1.169]
I20240502 14:24:56.982681 11916 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:24:57.011175 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:57.026104 12178 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:24:57.030289 13637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:57.032183 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:57.042198 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:24:57.078613 12045 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240502 14:24:57.325507 12133 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.65:36871, user_credentials={real_user=slave}} blocked reactor thread for 69209.4us
I20240502 14:24:58.386559 12068 consensus_queue.cc:237] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 182, Committed index: 182, Last appended: 1.182, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 183 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } }
I20240502 14:24:58.389958 13387 raft_consensus.cc:1261] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Refusing update from remote peer 36e66c7b1a03458b89a91c56a57df72a: Log matching property violated. Preceding OpId in replica: term: 1 index: 182. Preceding OpId from leader: term: 1 index: 183. (index mismatch)
I20240502 14:24:58.392820 13858 consensus_queue.cc:1035] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 183, Last known committed idx: 182, Time since last communication: 0.000s
I20240502 14:24:58.398964 14518 raft_consensus.cc:2932] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Committing config change with OpId 1.183: config changed from index 150 to 183, VOTER d9748c575342401bb80683c7e2c4e105 (127.5.107.67) evicted. New config: { opid_index: 183 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } }
I20240502 14:24:58.406523 13386 raft_consensus.cc:2932] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Committing config change with OpId 1.183: config changed from index 150 to 183, VOTER d9748c575342401bb80683c7e2c4e105 (127.5.107.67) evicted. New config: { opid_index: 183 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } }
I20240502 14:24:58.424155 11788 catalog_manager.cc:5484] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a reported cstate change: config changed from index 150 to 183, VOTER d9748c575342401bb80683c7e2c4e105 (127.5.107.67) evicted. New cstate: current_term: 1 leader_uuid: "36e66c7b1a03458b89a91c56a57df72a" committed_config { opid_index: 183 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } }
I20240502 14:24:58.520751 12178 tablet_service.cc:1514] Processing DeleteTablet for tablet 1427194e32d545f785df3e45137aa42e with delete_type TABLET_DATA_TOMBSTONED (TS d9748c575342401bb80683c7e2c4e105 not found in new config with opid_index 183) from {username='slave'} at 127.0.0.1:45972
I20240502 14:24:58.535537 14565 tablet_replica.cc:330] T 1427194e32d545f785df3e45137aa42e P d9748c575342401bb80683c7e2c4e105: stopping tablet replica
I20240502 14:24:58.549925 14565 raft_consensus.cc:2229] T 1427194e32d545f785df3e45137aa42e P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:24:58.570662 14565 raft_consensus.cc:2258] T 1427194e32d545f785df3e45137aa42e P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:24:58.588831 14565 ts_tablet_manager.cc:1875] T 1427194e32d545f785df3e45137aa42e P d9748c575342401bb80683c7e2c4e105: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:24:58.741072 14565 ts_tablet_manager.cc:1888] T 1427194e32d545f785df3e45137aa42e P d9748c575342401bb80683c7e2c4e105: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.182
I20240502 14:24:58.747656 14565 log.cc:1196] T 1427194e32d545f785df3e45137aa42e P d9748c575342401bb80683c7e2c4e105: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal/wals/1427194e32d545f785df3e45137aa42e
I20240502 14:24:58.750479 11777 catalog_manager.cc:4830] TS d9748c575342401bb80683c7e2c4e105 (127.5.107.67:45923): tablet 1427194e32d545f785df3e45137aa42e (table rebalance_test_table_4 [id=cc8c51baff5f4081aca1cf7908b85057]) successfully deleted
I20240502 14:24:59.348235 11916 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:24:59.409938 12045 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:24:59.461897 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:59.470470 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:24:59.518078 13637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:59.525457 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:24:59.543470 12178 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:24:59.790953 12068 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e05a3a4fb76d4c46908945ad0d838509"
dest_uuid: "36e66c7b1a03458b89a91c56a57df72a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56770
I20240502 14:24:59.791208 12068 raft_consensus.cc:595] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Received request to transfer leadership
W20240502 14:24:59.855337 12026 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32810: Service unavailable: leader transfer in progress
W20240502 14:24:59.860252 12026 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32810: Service unavailable: leader transfer in progress
W20240502 14:24:59.867053 12021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32810: Service unavailable: leader transfer in progress
W20240502 14:24:59.873549 12021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32810: Service unavailable: leader transfer in progress
W20240502 14:24:59.880751 12021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32810: Service unavailable: leader transfer in progress
W20240502 14:24:59.893563 12026 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32810: Service unavailable: leader transfer in progress
W20240502 14:24:59.911207 12026 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32810: Service unavailable: leader transfer in progress
W20240502 14:24:59.925488 12021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32810: Service unavailable: leader transfer in progress
W20240502 14:24:59.973146 12026 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32810: Service unavailable: leader transfer in progress
W20240502 14:24:59.986019 12026 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32810: Service unavailable: leader transfer in progress
W20240502 14:25:00.023476 12021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32810: Service unavailable: leader transfer in progress
W20240502 14:25:00.063464 12026 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32810: Service unavailable: leader transfer in progress
W20240502 14:25:00.078891 12021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32810: Service unavailable: leader transfer in progress
W20240502 14:25:00.096637 12021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32810: Service unavailable: leader transfer in progress
W20240502 14:25:00.117466 12026 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32810: Service unavailable: leader transfer in progress
W20240502 14:25:00.138542 12026 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32810: Service unavailable: leader transfer in progress
W20240502 14:25:00.157251 12026 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32810: Service unavailable: leader transfer in progress
I20240502 14:25:00.193058 14519 raft_consensus.cc:979] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a: : Instructing follower d9748c575342401bb80683c7e2c4e105 to start an election
I20240502 14:25:00.193914 14519 raft_consensus.cc:1067] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Signalling peer d9748c575342401bb80683c7e2c4e105 to start an election
W20240502 14:25:00.195511 12026 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32810: Service unavailable: leader transfer in progress
I20240502 14:25:00.208616 12199 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e05a3a4fb76d4c46908945ad0d838509"
dest_uuid: "d9748c575342401bb80683c7e2c4e105"
from {username='slave'} at 127.5.107.66:54597
I20240502 14:25:00.208840 12199 raft_consensus.cc:482] T e05a3a4fb76d4c46908945ad0d838509 P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:25:00.208961 12199 raft_consensus.cc:3037] T e05a3a4fb76d4c46908945ad0d838509 P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:25:00.221515 12021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32810: Service unavailable: leader transfer in progress
I20240502 14:25:00.240093 12199 raft_consensus.cc:504] T e05a3a4fb76d4c46908945ad0d838509 P d9748c575342401bb80683c7e2c4e105 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } }
I20240502 14:25:00.241828 12199 leader_election.cc:288] T e05a3a4fb76d4c46908945ad0d838509 P d9748c575342401bb80683c7e2c4e105 [CANDIDATE]: Term 2 election: Requested vote from peers 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445), b0d1c78721a746e99d0d3056f1e31371 (127.5.107.71:33799)
I20240502 14:25:00.244148 13655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05a3a4fb76d4c46908945ad0d838509" candidate_uuid: "d9748c575342401bb80683c7e2c4e105" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: true dest_uuid: "b0d1c78721a746e99d0d3056f1e31371"
W20240502 14:25:00.244307 12021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32810: Service unavailable: leader transfer in progress
I20240502 14:25:00.244494 13655 raft_consensus.cc:3037] T e05a3a4fb76d4c46908945ad0d838509 P b0d1c78721a746e99d0d3056f1e31371 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:25:00.248088 13655 raft_consensus.cc:2454] T e05a3a4fb76d4c46908945ad0d838509 P b0d1c78721a746e99d0d3056f1e31371 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9748c575342401bb80683c7e2c4e105 in term 2.
I20240502 14:25:00.249423 12134 leader_election.cc:302] T e05a3a4fb76d4c46908945ad0d838509 P d9748c575342401bb80683c7e2c4e105 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0d1c78721a746e99d0d3056f1e31371, d9748c575342401bb80683c7e2c4e105; no voters:
I20240502 14:25:00.250341 14526 raft_consensus.cc:2781] T e05a3a4fb76d4c46908945ad0d838509 P d9748c575342401bb80683c7e2c4e105 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:25:00.250582 14526 raft_consensus.cc:686] T e05a3a4fb76d4c46908945ad0d838509 P d9748c575342401bb80683c7e2c4e105 [term 2 LEADER]: Becoming Leader. State: Replica: d9748c575342401bb80683c7e2c4e105, State: Running, Role: LEADER
I20240502 14:25:00.250911 14526 consensus_queue.cc:237] T e05a3a4fb76d4c46908945ad0d838509 P d9748c575342401bb80683c7e2c4e105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } }
I20240502 14:25:00.254998 11788 catalog_manager.cc:5484] T e05a3a4fb76d4c46908945ad0d838509 P d9748c575342401bb80683c7e2c4e105 reported cstate change: term changed from 1 to 2, leader changed from 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66) to d9748c575342401bb80683c7e2c4e105 (127.5.107.67). New cstate: current_term: 2 leader_uuid: "d9748c575342401bb80683c7e2c4e105" committed_config { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } }
W20240502 14:25:00.267298 12021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32810: Service unavailable: leader transfer in progress
W20240502 14:25:00.291191 12021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32810: Service unavailable: leader transfer in progress
W20240502 14:25:00.317653 12026 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32810: Service unavailable: leader transfer in progress
W20240502 14:25:00.351629 12022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32810: Service unavailable: leader transfer in progress
I20240502 14:25:00.357820 12068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05a3a4fb76d4c46908945ad0d838509" candidate_uuid: "d9748c575342401bb80683c7e2c4e105" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: true dest_uuid: "36e66c7b1a03458b89a91c56a57df72a"
I20240502 14:25:00.358124 12068 raft_consensus.cc:3032] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:25:00.358238 12068 raft_consensus.cc:729] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 36e66c7b1a03458b89a91c56a57df72a, State: Running, Role: LEADER
I20240502 14:25:00.358537 12068 consensus_queue.cc:260] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } }
I20240502 14:25:00.358943 12068 raft_consensus.cc:3037] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:25:00.363601 12068 raft_consensus.cc:2454] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9748c575342401bb80683c7e2c4e105 in term 2.
W20240502 14:25:00.380017 12022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32810: Illegal state: replica 36e66c7b1a03458b89a91c56a57df72a is not leader of this config: current role FOLLOWER
I20240502 14:25:00.648195 12068 raft_consensus.cc:1261] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a [term 2 FOLLOWER]: Refusing update from remote peer d9748c575342401bb80683c7e2c4e105: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20240502 14:25:00.649209 14477 consensus_queue.cc:1035] T e05a3a4fb76d4c46908945ad0d838509 P d9748c575342401bb80683c7e2c4e105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20240502 14:25:00.666501 13656 raft_consensus.cc:1261] T e05a3a4fb76d4c46908945ad0d838509 P b0d1c78721a746e99d0d3056f1e31371 [term 2 FOLLOWER]: Refusing update from remote peer d9748c575342401bb80683c7e2c4e105: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20240502 14:25:00.680408 14137 consensus_queue.cc:1035] T e05a3a4fb76d4c46908945ad0d838509 P d9748c575342401bb80683c7e2c4e105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20240502 14:25:01.042948 12045 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:25:01.081455 13637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:01.139425 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:01.197295 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:01.279361 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:01.295018 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:01.322556 11916 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:01.996675 12195 consensus_queue.cc:237] T e05a3a4fb76d4c46908945ad0d838509 P d9748c575342401bb80683c7e2c4e105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 219, Committed index: 219, Last appended: 2.219, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } }
I20240502 14:25:01.998379 13657 raft_consensus.cc:1261] T e05a3a4fb76d4c46908945ad0d838509 P b0d1c78721a746e99d0d3056f1e31371 [term 2 FOLLOWER]: Refusing update from remote peer d9748c575342401bb80683c7e2c4e105: Log matching property violated. Preceding OpId in replica: term: 2 index: 219. Preceding OpId from leader: term: 2 index: 220. (index mismatch)
I20240502 14:25:01.999243 14477 consensus_queue.cc:1035] T e05a3a4fb76d4c46908945ad0d838509 P d9748c575342401bb80683c7e2c4e105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 220, Last known committed idx: 219, Time since last communication: 0.000s
I20240502 14:25:02.004537 13656 raft_consensus.cc:2932] T e05a3a4fb76d4c46908945ad0d838509 P b0d1c78721a746e99d0d3056f1e31371 [term 2 FOLLOWER]: Committing config change with OpId 2.220: config changed from index 140 to 220, VOTER 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66) evicted. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } }
I20240502 14:25:02.003931 14526 raft_consensus.cc:2932] T e05a3a4fb76d4c46908945ad0d838509 P d9748c575342401bb80683c7e2c4e105 [term 2 LEADER]: Committing config change with OpId 2.220: config changed from index 140 to 220, VOTER 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66) evicted. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } }
I20240502 14:25:02.027791 11788 catalog_manager.cc:5484] T e05a3a4fb76d4c46908945ad0d838509 P d9748c575342401bb80683c7e2c4e105 reported cstate change: config changed from index 140 to 220, VOTER 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66) evicted. New cstate: current_term: 2 leader_uuid: "d9748c575342401bb80683c7e2c4e105" committed_config { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } }
I20240502 14:25:02.141542 12045 tablet_service.cc:1514] Processing DeleteTablet for tablet e05a3a4fb76d4c46908945ad0d838509 with delete_type TABLET_DATA_TOMBSTONED (TS 36e66c7b1a03458b89a91c56a57df72a not found in new config with opid_index 220) from {username='slave'} at 127.0.0.1:36164
I20240502 14:25:02.145720 14739 tablet_replica.cc:330] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a: stopping tablet replica
I20240502 14:25:02.150981 14739 raft_consensus.cc:2229] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:25:02.151410 14739 raft_consensus.cc:2258] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:25:02.158463 14739 ts_tablet_manager.cc:1875] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:25:02.221661 14739 ts_tablet_manager.cc:1888] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.219
I20240502 14:25:02.222004 14739 log.cc:1196] T e05a3a4fb76d4c46908945ad0d838509 P 36e66c7b1a03458b89a91c56a57df72a: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/wals/e05a3a4fb76d4c46908945ad0d838509
I20240502 14:25:02.223426 11780 catalog_manager.cc:4830] TS 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445): tablet e05a3a4fb76d4c46908945ad0d838509 (table rebalance_test_table_0 [id=1ebc3641cd6e480c85c60dd430ccb365]) successfully deleted
I20240502 14:25:03.285205 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:03.314153 13637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:03.326994 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:03.453848 11916 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:03.461709 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:03.462627 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:03.475670 12045 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:25:04.082149 12068 consensus_queue.cc:237] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 258, Committed index: 258, Last appended: 1.258, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } }
I20240502 14:25:04.086658 13385 raft_consensus.cc:1261] T c3811b0d595f4292b9d96454deba8156 P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Refusing update from remote peer 36e66c7b1a03458b89a91c56a57df72a: Log matching property violated. Preceding OpId in replica: term: 1 index: 258. Preceding OpId from leader: term: 1 index: 259. (index mismatch)
I20240502 14:25:04.087543 13156 consensus_queue.cc:1035] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 259, Last known committed idx: 258, Time since last communication: 0.000s
I20240502 14:25:04.091496 14294 raft_consensus.cc:2932] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Committing config change with OpId 1.259: config changed from index 148 to 259, VOTER d9748c575342401bb80683c7e2c4e105 (127.5.107.67) evicted. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } }
I20240502 14:25:04.099406 11784 catalog_manager.cc:5484] T c3811b0d595f4292b9d96454deba8156 P 36e66c7b1a03458b89a91c56a57df72a reported cstate change: config changed from index 148 to 259, VOTER d9748c575342401bb80683c7e2c4e105 (127.5.107.67) evicted. New cstate: current_term: 1 leader_uuid: "36e66c7b1a03458b89a91c56a57df72a" committed_config { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } }
I20240502 14:25:04.111014 13387 raft_consensus.cc:2932] T c3811b0d595f4292b9d96454deba8156 P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Committing config change with OpId 1.259: config changed from index 148 to 259, VOTER d9748c575342401bb80683c7e2c4e105 (127.5.107.67) evicted. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } }
I20240502 14:25:04.229552 12178 tablet_service.cc:1514] Processing DeleteTablet for tablet c3811b0d595f4292b9d96454deba8156 with delete_type TABLET_DATA_TOMBSTONED (TS d9748c575342401bb80683c7e2c4e105 not found in new config with opid_index 259) from {username='slave'} at 127.0.0.1:45972
I20240502 14:25:04.350510 14824 tablet_replica.cc:330] T c3811b0d595f4292b9d96454deba8156 P d9748c575342401bb80683c7e2c4e105: stopping tablet replica
I20240502 14:25:04.350805 14824 raft_consensus.cc:2229] T c3811b0d595f4292b9d96454deba8156 P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:25:04.351203 14824 raft_consensus.cc:2258] T c3811b0d595f4292b9d96454deba8156 P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:25:04.352481 14824 ts_tablet_manager.cc:1875] T c3811b0d595f4292b9d96454deba8156 P d9748c575342401bb80683c7e2c4e105: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:25:04.358611 14824 ts_tablet_manager.cc:1888] T c3811b0d595f4292b9d96454deba8156 P d9748c575342401bb80683c7e2c4e105: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.258
I20240502 14:25:04.358754 14824 log.cc:1196] T c3811b0d595f4292b9d96454deba8156 P d9748c575342401bb80683c7e2c4e105: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal/wals/c3811b0d595f4292b9d96454deba8156
I20240502 14:25:04.360903 11777 catalog_manager.cc:4830] TS d9748c575342401bb80683c7e2c4e105 (127.5.107.67:45923): tablet c3811b0d595f4292b9d96454deba8156 (table rebalance_test_table_0 [id=1ebc3641cd6e480c85c60dd430ccb365]) successfully deleted
I20240502 14:25:05.558744 11916 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:05.639968 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:05.640897 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:05.644027 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:05.674034 13637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:05.681655 12045 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:25:05.758565 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:05.968858 12068 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e7732ff4fa2744db94d0f8cb888de8d8"
dest_uuid: "36e66c7b1a03458b89a91c56a57df72a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48364
I20240502 14:25:05.969069 12068 raft_consensus.cc:595] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Received request to transfer leadership
W20240502 14:25:06.015259 12021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:06.028101 12022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:06.036001 12022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:06.061661 12022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:06.070712 12022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:06.082795 12022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:06.096261 12022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:06.106633 12022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:06.122416 12021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:06.134965 12021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:06.158164 12021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:06.176677 12021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:06.193557 12021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:06.213886 12021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:06.234097 12021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:06.255152 12022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:06.277755 12021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:06.299906 12021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
I20240502 14:25:06.315515 13858 raft_consensus.cc:979] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a: : Instructing follower d9748c575342401bb80683c7e2c4e105 to start an election
I20240502 14:25:06.315734 13858 raft_consensus.cc:1067] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Signalling peer d9748c575342401bb80683c7e2c4e105 to start an election
I20240502 14:25:06.318701 12196 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e7732ff4fa2744db94d0f8cb888de8d8"
dest_uuid: "d9748c575342401bb80683c7e2c4e105"
from {username='slave'} at 127.5.107.66:54597
I20240502 14:25:06.319003 12196 raft_consensus.cc:482] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:25:06.319212 12196 raft_consensus.cc:3037] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:25:06.323253 12022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
I20240502 14:25:06.325356 12196 raft_consensus.cc:504] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } }
I20240502 14:25:06.327981 12196 leader_election.cc:288] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [CANDIDATE]: Term 2 election: Requested vote from peers 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445), b0d1c78721a746e99d0d3056f1e31371 (127.5.107.71:33799)
I20240502 14:25:06.328712 13657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7732ff4fa2744db94d0f8cb888de8d8" candidate_uuid: "d9748c575342401bb80683c7e2c4e105" candidate_term: 2 candidate_status { last_received { term: 1 index: 442 } } ignore_live_leader: true dest_uuid: "b0d1c78721a746e99d0d3056f1e31371"
I20240502 14:25:06.329231 13657 raft_consensus.cc:3037] T e7732ff4fa2744db94d0f8cb888de8d8 P b0d1c78721a746e99d0d3056f1e31371 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:25:06.332080 12068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7732ff4fa2744db94d0f8cb888de8d8" candidate_uuid: "d9748c575342401bb80683c7e2c4e105" candidate_term: 2 candidate_status { last_received { term: 1 index: 442 } } ignore_live_leader: true dest_uuid: "36e66c7b1a03458b89a91c56a57df72a"
I20240502 14:25:06.332399 12068 raft_consensus.cc:3032] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:25:06.332567 12068 raft_consensus.cc:729] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 36e66c7b1a03458b89a91c56a57df72a, State: Running, Role: LEADER
I20240502 14:25:06.332934 12068 consensus_queue.cc:260] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 442, Committed index: 442, Last appended: 1.442, Last appended by leader: 442, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } }
I20240502 14:25:06.333477 12068 raft_consensus.cc:3037] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:25:06.339735 12068 raft_consensus.cc:2454] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9748c575342401bb80683c7e2c4e105 in term 2.
W20240502 14:25:06.344920 12021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Illegal state: replica 36e66c7b1a03458b89a91c56a57df72a is not leader of this config: current role FOLLOWER
I20240502 14:25:06.347713 13657 raft_consensus.cc:2454] T e7732ff4fa2744db94d0f8cb888de8d8 P b0d1c78721a746e99d0d3056f1e31371 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9748c575342401bb80683c7e2c4e105 in term 2.
I20240502 14:25:06.341653 12135 leader_election.cc:302] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36e66c7b1a03458b89a91c56a57df72a, d9748c575342401bb80683c7e2c4e105; no voters:
I20240502 14:25:06.349908 14526 raft_consensus.cc:2781] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:25:06.350031 14526 raft_consensus.cc:686] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [term 2 LEADER]: Becoming Leader. State: Replica: d9748c575342401bb80683c7e2c4e105, State: Running, Role: LEADER
I20240502 14:25:06.350342 14526 consensus_queue.cc:237] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 442, Committed index: 442, Last appended: 1.442, Last appended by leader: 442, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } }
I20240502 14:25:06.354199 11788 catalog_manager.cc:5484] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 reported cstate change: term changed from 1 to 2, leader changed from 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66) to d9748c575342401bb80683c7e2c4e105 (127.5.107.67). New cstate: current_term: 2 leader_uuid: "d9748c575342401bb80683c7e2c4e105" committed_config { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } }
I20240502 14:25:06.461573 13657 raft_consensus.cc:1261] T e7732ff4fa2744db94d0f8cb888de8d8 P b0d1c78721a746e99d0d3056f1e31371 [term 2 FOLLOWER]: Refusing update from remote peer d9748c575342401bb80683c7e2c4e105: Log matching property violated. Preceding OpId in replica: term: 1 index: 442. Preceding OpId from leader: term: 2 index: 444. (index mismatch)
I20240502 14:25:06.462381 14526 consensus_queue.cc:1035] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 443, Last known committed idx: 442, Time since last communication: 0.000s
I20240502 14:25:06.464565 12068 raft_consensus.cc:1261] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a [term 2 FOLLOWER]: Refusing update from remote peer d9748c575342401bb80683c7e2c4e105: Log matching property violated. Preceding OpId in replica: term: 1 index: 442. Preceding OpId from leader: term: 2 index: 444. (index mismatch)
I20240502 14:25:06.468608 14477 consensus_queue.cc:1035] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 443, Last known committed idx: 442, Time since last communication: 0.000s
I20240502 14:25:06.635143 14641 mvcc.cc:205] Tried to move back new op lower bound from 7023246976855265280 to 7023246976413270016. Current Snapshot: MvccSnapshot[applied={T|T < 7023246976855265280}]
I20240502 14:25:06.904973 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:07.042637 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:07.094137 12178 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:25:07.095280 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:07.240288 12045 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:25:07.276705 13637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:07.471462 11916 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:25:08.044548 12135 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.70:33951, user_credentials={real_user=slave}} blocked reactor thread for 61274.1us
I20240502 14:25:08.486733 12195 consensus_queue.cc:237] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 487, Committed index: 487, Last appended: 2.487, Last appended by leader: 442, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 488 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } }
I20240502 14:25:08.493602 13657 raft_consensus.cc:1261] T e7732ff4fa2744db94d0f8cb888de8d8 P b0d1c78721a746e99d0d3056f1e31371 [term 2 FOLLOWER]: Refusing update from remote peer d9748c575342401bb80683c7e2c4e105: Log matching property violated. Preceding OpId in replica: term: 2 index: 487. Preceding OpId from leader: term: 2 index: 489. (index mismatch)
I20240502 14:25:08.501240 14980 consensus_queue.cc:1035] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 488, Last known committed idx: 487, Time since last communication: 0.000s
I20240502 14:25:08.509358 14529 raft_consensus.cc:2932] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [term 2 LEADER]: Committing config change with OpId 2.488: config changed from index 128 to 488, VOTER 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66) evicted. New config: { opid_index: 488 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } }
I20240502 14:25:08.514536 13655 raft_consensus.cc:2932] T e7732ff4fa2744db94d0f8cb888de8d8 P b0d1c78721a746e99d0d3056f1e31371 [term 2 FOLLOWER]: Committing config change with OpId 2.488: config changed from index 128 to 488, VOTER 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66) evicted. New config: { opid_index: 488 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } }
I20240502 14:25:08.522567 11784 catalog_manager.cc:5484] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 reported cstate change: config changed from index 128 to 488, VOTER 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66) evicted. New cstate: current_term: 2 leader_uuid: "d9748c575342401bb80683c7e2c4e105" committed_config { opid_index: 488 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } }
I20240502 14:25:08.612305 12045 tablet_service.cc:1514] Processing DeleteTablet for tablet e7732ff4fa2744db94d0f8cb888de8d8 with delete_type TABLET_DATA_TOMBSTONED (TS 36e66c7b1a03458b89a91c56a57df72a not found in new config with opid_index 488) from {username='slave'} at 127.0.0.1:36164
I20240502 14:25:08.612967 15007 tablet_replica.cc:330] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a: stopping tablet replica
I20240502 14:25:08.616344 15007 raft_consensus.cc:2229] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:25:08.616608 15007 raft_consensus.cc:2258] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:25:08.618608 15007 ts_tablet_manager.cc:1875] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:25:08.640504 15007 ts_tablet_manager.cc:1888] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.487
I20240502 14:25:08.641453 15007 log.cc:1196] T e7732ff4fa2744db94d0f8cb888de8d8 P 36e66c7b1a03458b89a91c56a57df72a: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/wals/e7732ff4fa2744db94d0f8cb888de8d8
I20240502 14:25:08.642830 11780 catalog_manager.cc:4830] TS 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445): tablet e7732ff4fa2744db94d0f8cb888de8d8 (table rebalance_test_table_2 [id=ec56bc8cfbc04f26b92384c442eac5b8]) successfully deleted
I20240502 14:25:09.653950 11916 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:09.714748 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:09.718366 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:09.751116 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:09.767637 13637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:09.790763 12045 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:25:09.821631 12178 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:25:10.017719 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.65:36871, user_credentials={real_user=slave}} blocked reactor thread for 104055us
I20240502 14:25:10.898874 11936 consensus_queue.cc:237] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 518, Committed index: 518, Last appended: 1.518, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 519 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } }
I20240502 14:25:10.913585 13383 raft_consensus.cc:1261] T 877f199681534650bc1d247cf7608f5f P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Refusing update from remote peer 0e41abe35d8f40a2a01953eea1c5d092: Log matching property violated. Preceding OpId in replica: term: 1 index: 518. Preceding OpId from leader: term: 1 index: 519. (index mismatch)
I20240502 14:25:10.914263 14955 consensus_queue.cc:1035] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 519, Last known committed idx: 518, Time since last communication: 0.000s
I20240502 14:25:10.916734 14955 raft_consensus.cc:2932] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 LEADER]: Committing config change with OpId 1.519: config changed from index 116 to 519, VOTER d9748c575342401bb80683c7e2c4e105 (127.5.107.67) evicted. New config: { opid_index: 519 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } }
I20240502 14:25:10.924762 13385 raft_consensus.cc:2932] T 877f199681534650bc1d247cf7608f5f P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Committing config change with OpId 1.519: config changed from index 116 to 519, VOTER d9748c575342401bb80683c7e2c4e105 (127.5.107.67) evicted. New config: { opid_index: 519 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } }
I20240502 14:25:10.944561 11784 catalog_manager.cc:5484] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 reported cstate change: config changed from index 116 to 519, VOTER d9748c575342401bb80683c7e2c4e105 (127.5.107.67) evicted. New cstate: current_term: 1 leader_uuid: "0e41abe35d8f40a2a01953eea1c5d092" committed_config { opid_index: 519 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } }
I20240502 14:25:11.162734 12178 tablet_service.cc:1514] Processing DeleteTablet for tablet 877f199681534650bc1d247cf7608f5f with delete_type TABLET_DATA_TOMBSTONED (TS d9748c575342401bb80683c7e2c4e105 not found in new config with opid_index 519) from {username='slave'} at 127.0.0.1:45972
I20240502 14:25:11.177735 15106 tablet_replica.cc:330] T 877f199681534650bc1d247cf7608f5f P d9748c575342401bb80683c7e2c4e105: stopping tablet replica
I20240502 14:25:11.178009 15106 raft_consensus.cc:2229] T 877f199681534650bc1d247cf7608f5f P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:25:11.178305 15106 raft_consensus.cc:2258] T 877f199681534650bc1d247cf7608f5f P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:25:11.184056 15106 ts_tablet_manager.cc:1875] T 877f199681534650bc1d247cf7608f5f P d9748c575342401bb80683c7e2c4e105: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:25:11.190589 15106 ts_tablet_manager.cc:1888] T 877f199681534650bc1d247cf7608f5f P d9748c575342401bb80683c7e2c4e105: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.518
I20240502 14:25:11.190757 15106 log.cc:1196] T 877f199681534650bc1d247cf7608f5f P d9748c575342401bb80683c7e2c4e105: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal/wals/877f199681534650bc1d247cf7608f5f
I20240502 14:25:11.211818 11777 catalog_manager.cc:4830] TS d9748c575342401bb80683c7e2c4e105 (127.5.107.67:45923): tablet 877f199681534650bc1d247cf7608f5f (table rebalance_test_table_3 [id=423ad8739bd34e329c45d60dce8f9687]) successfully deleted
I20240502 14:25:11.937844 11916 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:11.945410 12045 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:25:12.075554 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:12.103861 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:12.104763 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:12.138339 13637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:12.240056 12178 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:25:12.613601 12195 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bb1b842f76d74f6e996e48c245deda9b"
dest_uuid: "d9748c575342401bb80683c7e2c4e105"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37008
I20240502 14:25:12.613806 12195 raft_consensus.cc:595] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 [term 1 LEADER]: Received request to transfer leadership
W20240502 14:25:12.624058 12158 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45824: Service unavailable: leader transfer in progress
W20240502 14:25:12.628875 12158 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45824: Service unavailable: leader transfer in progress
I20240502 14:25:12.638152 14529 raft_consensus.cc:979] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105: : Instructing follower 0e41abe35d8f40a2a01953eea1c5d092 to start an election
W20240502 14:25:12.638221 12158 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45824: Service unavailable: leader transfer in progress
I20240502 14:25:12.638271 14529 raft_consensus.cc:1067] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 [term 1 LEADER]: Signalling peer 0e41abe35d8f40a2a01953eea1c5d092 to start an election
I20240502 14:25:12.645776 11935 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bb1b842f76d74f6e996e48c245deda9b"
dest_uuid: "0e41abe35d8f40a2a01953eea1c5d092"
from {username='slave'} at 127.5.107.67:34343
I20240502 14:25:12.646046 11935 raft_consensus.cc:482] T bb1b842f76d74f6e996e48c245deda9b P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:25:12.646193 11935 raft_consensus.cc:3037] T bb1b842f76d74f6e996e48c245deda9b P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:25:12.649758 12158 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45824: Service unavailable: leader transfer in progress
I20240502 14:25:12.651544 11935 raft_consensus.cc:504] T bb1b842f76d74f6e996e48c245deda9b P 0e41abe35d8f40a2a01953eea1c5d092 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:25:12.658206 11935 leader_election.cc:288] T bb1b842f76d74f6e996e48c245deda9b P 0e41abe35d8f40a2a01953eea1c5d092 [CANDIDATE]: Term 2 election: Requested vote from peers d9748c575342401bb80683c7e2c4e105 (127.5.107.67:45923), 375c6d5bdf924866b69b0e6667732692 (127.5.107.70:33951)
I20240502 14:25:12.658452 12195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb1b842f76d74f6e996e48c245deda9b" candidate_uuid: "0e41abe35d8f40a2a01953eea1c5d092" candidate_term: 2 candidate_status { last_received { term: 1 index: 453 } } ignore_live_leader: true dest_uuid: "d9748c575342401bb80683c7e2c4e105"
I20240502 14:25:12.658638 12195 raft_consensus.cc:3032] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:25:12.658710 12195 raft_consensus.cc:729] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d9748c575342401bb80683c7e2c4e105, State: Running, Role: LEADER
I20240502 14:25:12.659034 12195 consensus_queue.cc:260] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 453, Committed index: 453, Last appended: 1.453, Last appended by leader: 453, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:25:12.659384 12195 raft_consensus.cc:3037] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:25:12.662063 12195 raft_consensus.cc:2454] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e41abe35d8f40a2a01953eea1c5d092 in term 2.
I20240502 14:25:12.662510 11868 leader_election.cc:302] T bb1b842f76d74f6e996e48c245deda9b P 0e41abe35d8f40a2a01953eea1c5d092 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e41abe35d8f40a2a01953eea1c5d092, d9748c575342401bb80683c7e2c4e105; no voters:
I20240502 14:25:12.662850 14955 raft_consensus.cc:2781] T bb1b842f76d74f6e996e48c245deda9b P 0e41abe35d8f40a2a01953eea1c5d092 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:25:12.662986 14955 raft_consensus.cc:686] T bb1b842f76d74f6e996e48c245deda9b P 0e41abe35d8f40a2a01953eea1c5d092 [term 2 LEADER]: Becoming Leader. State: Replica: 0e41abe35d8f40a2a01953eea1c5d092, State: Running, Role: LEADER
I20240502 14:25:12.663257 14955 consensus_queue.cc:237] T bb1b842f76d74f6e996e48c245deda9b P 0e41abe35d8f40a2a01953eea1c5d092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 453, Committed index: 453, Last appended: 1.453, Last appended by leader: 453, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:25:12.667917 11784 catalog_manager.cc:5484] T bb1b842f76d74f6e996e48c245deda9b P 0e41abe35d8f40a2a01953eea1c5d092 reported cstate change: term changed from 1 to 2, leader changed from d9748c575342401bb80683c7e2c4e105 (127.5.107.67) to 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65). New cstate: current_term: 2 leader_uuid: "0e41abe35d8f40a2a01953eea1c5d092" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } } }
W20240502 14:25:12.669723 12158 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45824: Illegal state: replica d9748c575342401bb80683c7e2c4e105 is not leader of this config: current role FOLLOWER
I20240502 14:25:12.765092 13520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb1b842f76d74f6e996e48c245deda9b" candidate_uuid: "0e41abe35d8f40a2a01953eea1c5d092" candidate_term: 2 candidate_status { last_received { term: 1 index: 453 } } ignore_live_leader: true dest_uuid: "375c6d5bdf924866b69b0e6667732692"
I20240502 14:25:12.765419 13520 raft_consensus.cc:3037] T bb1b842f76d74f6e996e48c245deda9b P 375c6d5bdf924866b69b0e6667732692 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:25:12.768055 13520 raft_consensus.cc:2454] T bb1b842f76d74f6e996e48c245deda9b P 375c6d5bdf924866b69b0e6667732692 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e41abe35d8f40a2a01953eea1c5d092 in term 2.
I20240502 14:25:12.840142 12195 raft_consensus.cc:1261] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 [term 2 FOLLOWER]: Refusing update from remote peer 0e41abe35d8f40a2a01953eea1c5d092: Log matching property violated. Preceding OpId in replica: term: 1 index: 453. Preceding OpId from leader: term: 2 index: 455. (index mismatch)
I20240502 14:25:12.840167 13518 raft_consensus.cc:1261] T bb1b842f76d74f6e996e48c245deda9b P 375c6d5bdf924866b69b0e6667732692 [term 2 FOLLOWER]: Refusing update from remote peer 0e41abe35d8f40a2a01953eea1c5d092: Log matching property violated. Preceding OpId in replica: term: 1 index: 453. Preceding OpId from leader: term: 2 index: 455. (index mismatch)
I20240502 14:25:12.840667 14955 consensus_queue.cc:1035] T bb1b842f76d74f6e996e48c245deda9b P 0e41abe35d8f40a2a01953eea1c5d092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 454, Last known committed idx: 453, Time since last communication: 0.000s
I20240502 14:25:12.841547 14955 consensus_queue.cc:1035] T bb1b842f76d74f6e996e48c245deda9b P 0e41abe35d8f40a2a01953eea1c5d092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 454, Last known committed idx: 453, Time since last communication: 0.000s
I20240502 14:25:12.844516 15058 mvcc.cc:205] Tried to move back new op lower bound from 7023247002986213376 to 7023247002271092736. Current Snapshot: MvccSnapshot[applied={T|T < 7023247002986213376}]
W20240502 14:25:13.580046 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.65:36871, user_credentials={real_user=slave}} blocked reactor thread for 81635.7us
I20240502 14:25:13.800792 11916 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:25:13.880185 12045 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:25:14.174466 13637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:14.216869 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:14.226051 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:14.266775 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:14.433108 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:14.862339 11930 consensus_queue.cc:237] T bb1b842f76d74f6e996e48c245deda9b P 0e41abe35d8f40a2a01953eea1c5d092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 496, Committed index: 496, Last appended: 2.496, Last appended by leader: 453, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 497 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:25:14.869675 13520 raft_consensus.cc:1261] T bb1b842f76d74f6e996e48c245deda9b P 375c6d5bdf924866b69b0e6667732692 [term 2 FOLLOWER]: Refusing update from remote peer 0e41abe35d8f40a2a01953eea1c5d092: Log matching property violated. Preceding OpId in replica: term: 2 index: 496. Preceding OpId from leader: term: 2 index: 497. (index mismatch)
I20240502 14:25:14.870801 15190 consensus_queue.cc:1035] T bb1b842f76d74f6e996e48c245deda9b P 0e41abe35d8f40a2a01953eea1c5d092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 497, Last known committed idx: 496, Time since last communication: 0.000s
I20240502 14:25:14.876545 15190 raft_consensus.cc:2932] T bb1b842f76d74f6e996e48c245deda9b P 0e41abe35d8f40a2a01953eea1c5d092 [term 2 LEADER]: Committing config change with OpId 2.497: config changed from index 106 to 497, VOTER d9748c575342401bb80683c7e2c4e105 (127.5.107.67) evicted. New config: { opid_index: 497 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } } }
I20240502 14:25:14.880178 13518 raft_consensus.cc:2932] T bb1b842f76d74f6e996e48c245deda9b P 375c6d5bdf924866b69b0e6667732692 [term 2 FOLLOWER]: Committing config change with OpId 2.497: config changed from index 106 to 497, VOTER d9748c575342401bb80683c7e2c4e105 (127.5.107.67) evicted. New config: { opid_index: 497 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } } }
I20240502 14:25:14.888165 11784 catalog_manager.cc:5484] T bb1b842f76d74f6e996e48c245deda9b P 375c6d5bdf924866b69b0e6667732692 reported cstate change: config changed from index 106 to 497, VOTER d9748c575342401bb80683c7e2c4e105 (127.5.107.67) evicted. New cstate: current_term: 2 leader_uuid: "0e41abe35d8f40a2a01953eea1c5d092" committed_config { opid_index: 497 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } } }
I20240502 14:25:15.097759 12178 tablet_service.cc:1514] Processing DeleteTablet for tablet bb1b842f76d74f6e996e48c245deda9b with delete_type TABLET_DATA_TOMBSTONED (TS d9748c575342401bb80683c7e2c4e105 not found in new config with opid_index 497) from {username='slave'} at 127.0.0.1:45972
I20240502 14:25:15.122758 15278 tablet_replica.cc:330] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105: stopping tablet replica
I20240502 14:25:15.123111 15278 raft_consensus.cc:2229] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:25:15.123384 15278 raft_consensus.cc:2258] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:25:15.124388 15278 ts_tablet_manager.cc:1875] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:25:15.133718 15278 ts_tablet_manager.cc:1888] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.496
I20240502 14:25:15.141327 15278 log.cc:1196] T bb1b842f76d74f6e996e48c245deda9b P d9748c575342401bb80683c7e2c4e105: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal/wals/bb1b842f76d74f6e996e48c245deda9b
I20240502 14:25:15.143102 11777 catalog_manager.cc:4830] TS d9748c575342401bb80683c7e2c4e105 (127.5.107.67:45923): tablet bb1b842f76d74f6e996e48c245deda9b (table rebalance_test_table_1 [id=fe50532b3c3d46b7bdae21e985f105f6]) successfully deleted
I20240502 14:25:16.017197 11916 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:25:16.106194 12045 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:25:16.191722 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:16.229631 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:16.300773 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:16.310012 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:16.373842 13637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 14:25:16.714265 12134 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.71:33799, user_credentials={real_user=slave}} blocked reactor thread for 65408.9us
I20240502 14:25:16.813140 13522 consensus_queue.cc:237] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } }
I20240502 14:25:16.849704 13249 raft_consensus.cc:1261] T f279550142c648d89b81feee9740f43e P 93c82e87d8af478292ffe1a3d3986049 [term 2 FOLLOWER]: Refusing update from remote peer 375c6d5bdf924866b69b0e6667732692: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:25:16.853502 14418 consensus_queue.cc:1035] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:25:16.978374 15372 raft_consensus.cc:2932] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } } }
I20240502 14:25:16.978724 13249 raft_consensus.cc:2932] T f279550142c648d89b81feee9740f43e P 93c82e87d8af478292ffe1a3d3986049 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } } }
I20240502 14:25:16.992995 11792 catalog_manager.cc:5484] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 reported cstate change: config changed from index 3 to 4, VOTER 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65) evicted. New cstate: current_term: 2 leader_uuid: "375c6d5bdf924866b69b0e6667732692" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } } }
I20240502 14:25:17.127960 11916 tablet_service.cc:1514] Processing DeleteTablet for tablet f279550142c648d89b81feee9740f43e with delete_type TABLET_DATA_TOMBSTONED (TS 0e41abe35d8f40a2a01953eea1c5d092 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42830
I20240502 14:25:17.128862 15386 tablet_replica.cc:330] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092: stopping tablet replica
I20240502 14:25:17.129474 15386 raft_consensus.cc:2229] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:25:17.129745 15386 raft_consensus.cc:2258] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:25:17.131695 15386 ts_tablet_manager.cc:1875] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:25:17.141789 15386 ts_tablet_manager.cc:1888] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240502 14:25:17.142163 15386 log.cc:1196] T f279550142c648d89b81feee9740f43e P 0e41abe35d8f40a2a01953eea1c5d092: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal/wals/f279550142c648d89b81feee9740f43e
I20240502 14:25:17.144234 11778 catalog_manager.cc:4830] TS 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871): tablet f279550142c648d89b81feee9740f43e (table TestTable [id=06ce4d13691e489389354201980c0142]) successfully deleted
W20240502 14:25:18.042801 12133 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.65:36871, user_credentials={real_user=slave}} blocked reactor thread for 94658.2us
I20240502 14:25:18.372005 11916 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:25:18.445174 13637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:18.478479 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:18.499773 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:18.503996 12045 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:25:18.572144 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:18.644429 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:25:19.015739 11868 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.69:33193, user_credentials={real_user=slave}} blocked reactor thread for 69710.2us
I20240502 14:25:19.373749 12200 consensus_queue.cc:237] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 721, Committed index: 721, Last appended: 1.722, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 723 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } }
I20240502 14:25:19.381382 13655 raft_consensus.cc:1261] T 71d8679988d0444c9e0434d7e6f142e8 P b0d1c78721a746e99d0d3056f1e31371 [term 1 FOLLOWER]: Refusing update from remote peer d9748c575342401bb80683c7e2c4e105: Log matching property violated. Preceding OpId in replica: term: 1 index: 721. Preceding OpId from leader: term: 1 index: 723. (index mismatch)
I20240502 14:25:19.386472 15289 consensus_queue.cc:1035] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 723, Last known committed idx: 721, Time since last communication: 0.000s
I20240502 14:25:19.392845 13655 raft_consensus.cc:1203] T 71d8679988d0444c9e0434d7e6f142e8 P b0d1c78721a746e99d0d3056f1e31371 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.721->[1.722-1.723] Dedup: 1.722->[1.723-1.723]
I20240502 14:25:19.397590 15412 raft_consensus.cc:2932] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105 [term 1 LEADER]: Committing config change with OpId 1.723: config changed from index 85 to 723, VOTER 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65) evicted. New config: { opid_index: 723 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } }
I20240502 14:25:19.405257 13655 raft_consensus.cc:2932] T 71d8679988d0444c9e0434d7e6f142e8 P b0d1c78721a746e99d0d3056f1e31371 [term 1 FOLLOWER]: Committing config change with OpId 1.723: config changed from index 85 to 723, VOTER 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65) evicted. New config: { opid_index: 723 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } }
I20240502 14:25:19.405970 11784 catalog_manager.cc:5484] T 71d8679988d0444c9e0434d7e6f142e8 P d9748c575342401bb80683c7e2c4e105 reported cstate change: config changed from index 85 to 723, VOTER 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65) evicted. New cstate: current_term: 1 leader_uuid: "d9748c575342401bb80683c7e2c4e105" committed_config { opid_index: 723 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } }
I20240502 14:25:19.594429 11916 tablet_service.cc:1514] Processing DeleteTablet for tablet 71d8679988d0444c9e0434d7e6f142e8 with delete_type TABLET_DATA_TOMBSTONED (TS 0e41abe35d8f40a2a01953eea1c5d092 not found in new config with opid_index 723) from {username='slave'} at 127.0.0.1:42830
I20240502 14:25:19.613663 15487 tablet_replica.cc:330] T 71d8679988d0444c9e0434d7e6f142e8 P 0e41abe35d8f40a2a01953eea1c5d092: stopping tablet replica
I20240502 14:25:19.613940 15487 raft_consensus.cc:2229] T 71d8679988d0444c9e0434d7e6f142e8 P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:25:19.614162 15487 pending_rounds.cc:62] T 71d8679988d0444c9e0434d7e6f142e8 P 0e41abe35d8f40a2a01953eea1c5d092: Trying to abort 1 pending ops.
I20240502 14:25:19.614332 15487 pending_rounds.cc:68] T 71d8679988d0444c9e0434d7e6f142e8 P 0e41abe35d8f40a2a01953eea1c5d092: Aborting op as it isn't in flight: id { term: 1 index: 722 } timestamp: 7023247029721870336 op_type: WRITE_OP write_request { tablet_id: "71d8679988d0444c9e0434d7e6f142e8" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\230\033\347\023\027\235\177\254\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""ca67431102c82cda" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023247029662318592 authz_token { token_data: "\010\331\306\316\261\006\"3\n\005slave\022*\n 423ad8739bd34e329c45d60dce8f9687\020\001\030\001 \001(\001" signature: "<redacted>""i\217\253\301\237\270\213\261\014\225h\026\t\203F\3358.\362+\226\304B\022\216QCW}\237\355\344e\351\376\252\276\177\022\037\031?/AQW\215/\250\t\345\364(\315\340x\347Av-\266\037:\367" signing_key_seq_num: 0 } } request_id { client_id: "9926769d70d14669a4f2473ddeddb20c" seq_no: 2043 first_incomplete_seq_no: 2043 attempt_no: 0 }
I20240502 14:25:19.615188 15487 raft_consensus.cc:2258] T 71d8679988d0444c9e0434d7e6f142e8 P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:25:19.618014 15487 ts_tablet_manager.cc:1875] T 71d8679988d0444c9e0434d7e6f142e8 P 0e41abe35d8f40a2a01953eea1c5d092: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:25:19.649508 15487 ts_tablet_manager.cc:1888] T 71d8679988d0444c9e0434d7e6f142e8 P 0e41abe35d8f40a2a01953eea1c5d092: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.722
I20240502 14:25:19.650301 15487 log.cc:1196] T 71d8679988d0444c9e0434d7e6f142e8 P 0e41abe35d8f40a2a01953eea1c5d092: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal/wals/71d8679988d0444c9e0434d7e6f142e8
I20240502 14:25:19.651330 11778 catalog_manager.cc:4830] TS 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871): tablet 71d8679988d0444c9e0434d7e6f142e8 (table rebalance_test_table_3 [id=423ad8739bd34e329c45d60dce8f9687]) successfully deleted
I20240502 14:25:20.393536 11916 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:25:20.579092 12045 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:25:20.712253 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:20.714623 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:20.875676 13637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:20.956398 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:20.961097 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:21.506316 12068 consensus_queue.cc:237] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 596, Committed index: 596, Last appended: 1.596, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 597 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } }
I20240502 14:25:21.519271 13384 raft_consensus.cc:1261] T 64e9d77a5b9343489db94f4988180213 P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Refusing update from remote peer 36e66c7b1a03458b89a91c56a57df72a: Log matching property violated. Preceding OpId in replica: term: 1 index: 596. Preceding OpId from leader: term: 1 index: 597. (index mismatch)
I20240502 14:25:21.521658 13802 consensus_queue.cc:1035] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 597, Last known committed idx: 596, Time since last communication: 0.000s
I20240502 14:25:21.532658 15421 raft_consensus.cc:2932] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Committing config change with OpId 1.597: config changed from index 63 to 597, VOTER 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65) evicted. New config: { opid_index: 597 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } }
I20240502 14:25:21.539983 11792 catalog_manager.cc:5484] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a reported cstate change: config changed from index 63 to 597, VOTER 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65) evicted. New cstate: current_term: 1 leader_uuid: "36e66c7b1a03458b89a91c56a57df72a" committed_config { opid_index: 597 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } }
I20240502 14:25:21.552253 13387 raft_consensus.cc:2932] T 64e9d77a5b9343489db94f4988180213 P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Committing config change with OpId 1.597: config changed from index 63 to 597, VOTER 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65) evicted. New config: { opid_index: 597 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } }
I20240502 14:25:21.709473 11916 tablet_service.cc:1514] Processing DeleteTablet for tablet 64e9d77a5b9343489db94f4988180213 with delete_type TABLET_DATA_TOMBSTONED (TS 0e41abe35d8f40a2a01953eea1c5d092 not found in new config with opid_index 597) from {username='slave'} at 127.0.0.1:42830
I20240502 14:25:21.710538 15588 tablet_replica.cc:330] T 64e9d77a5b9343489db94f4988180213 P 0e41abe35d8f40a2a01953eea1c5d092: stopping tablet replica
I20240502 14:25:21.710842 15588 raft_consensus.cc:2229] T 64e9d77a5b9343489db94f4988180213 P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:25:21.711799 15588 raft_consensus.cc:2258] T 64e9d77a5b9343489db94f4988180213 P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:25:21.714332 15588 ts_tablet_manager.cc:1875] T 64e9d77a5b9343489db94f4988180213 P 0e41abe35d8f40a2a01953eea1c5d092: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:25:21.724800 15588 ts_tablet_manager.cc:1888] T 64e9d77a5b9343489db94f4988180213 P 0e41abe35d8f40a2a01953eea1c5d092: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.596
I20240502 14:25:21.725397 15588 log.cc:1196] T 64e9d77a5b9343489db94f4988180213 P 0e41abe35d8f40a2a01953eea1c5d092: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal/wals/64e9d77a5b9343489db94f4988180213
I20240502 14:25:21.726256 11778 catalog_manager.cc:4830] TS 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871): tablet 64e9d77a5b9343489db94f4988180213 (table rebalance_test_table_1 [id=fe50532b3c3d46b7bdae21e985f105f6]) successfully deleted
I20240502 14:25:22.885502 11916 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:25:22.930145 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:22.941426 12045 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:25:22.960726 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:22.988435 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:23.137801 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:23.244665 13637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 14:25:23.805285 12000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.65:36871, user_credentials={real_user=slave}} blocked reactor thread for 122966us
W20240502 14:25:23.836598 12022 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.000s sys 0.011s
W20240502 14:25:23.836977 12022 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.508s user 0.000s sys 0.011s
W20240502 14:25:23.855371 11988 kernel_stack_watchdog.cc:198] Thread 12026 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:637 for 347ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f0b093f5980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7f0b06063d11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7f0b060847ab kudu::ThreadPool::CreateThread() at ??:0
@ 0x7f0b060803ad kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7f0b0607e84f kudu::ThreadPoolToken::Submit() at ??:0
@ 0x7f0b12bea32a kudu::tablet::OpDriver::ExecuteAsync() at ??:0
@ 0x7f0b12b90c00 kudu::tablet::TabletReplica::SubmitWrite() at ??:0
@ 0x7f0b1556aaa1 kudu::tserver::TabletServiceImpl::Write() at ??:0
@ 0x7f0b0fd0fdc7 kudu::tserver::TabletServerServiceIf::TabletServerServiceIf()::$_3::operator()() at ??:0
@ 0x7f0b0fd0faf6 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f0b0cd4804f std::function<>::operator()() at ??:0
@ 0x7f0b0cd46f2f kudu::rpc::GeneratedServiceIf::Handle() at ??:0
@ 0x7f0b0cd4d406 kudu::rpc::ServicePool::RunThread() at ??:0
W20240502 14:25:23.911129 12026 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.104s sys 0.248s
W20240502 14:25:23.911262 12026 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.583s user 0.104s sys 0.248s
I20240502 14:25:24.230536 12068 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4987edf6ce4d44d18162ff06e3f74e23"
dest_uuid: "36e66c7b1a03458b89a91c56a57df72a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55528
I20240502 14:25:24.230744 12068 raft_consensus.cc:595] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Received request to transfer leadership
I20240502 14:25:24.277462 13802 raft_consensus.cc:979] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a: : Instructing follower 0e41abe35d8f40a2a01953eea1c5d092 to start an election
I20240502 14:25:24.277648 13802 raft_consensus.cc:1067] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Signalling peer 0e41abe35d8f40a2a01953eea1c5d092 to start an election
I20240502 14:25:24.293352 11931 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4987edf6ce4d44d18162ff06e3f74e23"
dest_uuid: "0e41abe35d8f40a2a01953eea1c5d092"
from {username='slave'} at 127.5.107.66:60283
I20240502 14:25:24.293524 11931 raft_consensus.cc:482] T 4987edf6ce4d44d18162ff06e3f74e23 P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:25:24.293599 11931 raft_consensus.cc:3037] T 4987edf6ce4d44d18162ff06e3f74e23 P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:25:24.296074 11931 raft_consensus.cc:504] T 4987edf6ce4d44d18162ff06e3f74e23 P 0e41abe35d8f40a2a01953eea1c5d092 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } }
I20240502 14:25:24.302362 11931 leader_election.cc:288] T 4987edf6ce4d44d18162ff06e3f74e23 P 0e41abe35d8f40a2a01953eea1c5d092 [CANDIDATE]: Term 2 election: Requested vote from peers 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445), 93c82e87d8af478292ffe1a3d3986049 (127.5.107.68:37273)
W20240502 14:25:24.335512 12022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32810: Service unavailable: leader transfer in progress
I20240502 14:25:24.339210 12068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4987edf6ce4d44d18162ff06e3f74e23" candidate_uuid: "0e41abe35d8f40a2a01953eea1c5d092" candidate_term: 2 candidate_status { last_received { term: 1 index: 523 } } ignore_live_leader: true dest_uuid: "36e66c7b1a03458b89a91c56a57df72a"
I20240502 14:25:24.339468 12068 raft_consensus.cc:3032] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:25:24.339573 12068 raft_consensus.cc:729] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 36e66c7b1a03458b89a91c56a57df72a, State: Running, Role: LEADER
I20240502 14:25:24.339828 12068 consensus_queue.cc:260] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 523, Committed index: 523, Last appended: 1.523, Last appended by leader: 523, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } }
I20240502 14:25:24.340229 12068 raft_consensus.cc:3037] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:25:24.343429 12068 raft_consensus.cc:2454] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e41abe35d8f40a2a01953eea1c5d092 in term 2.
I20240502 14:25:24.343886 11871 leader_election.cc:302] T 4987edf6ce4d44d18162ff06e3f74e23 P 0e41abe35d8f40a2a01953eea1c5d092 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e41abe35d8f40a2a01953eea1c5d092, 36e66c7b1a03458b89a91c56a57df72a; no voters:
I20240502 14:25:24.344139 15662 raft_consensus.cc:2781] T 4987edf6ce4d44d18162ff06e3f74e23 P 0e41abe35d8f40a2a01953eea1c5d092 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:25:24.344266 15662 raft_consensus.cc:686] T 4987edf6ce4d44d18162ff06e3f74e23 P 0e41abe35d8f40a2a01953eea1c5d092 [term 2 LEADER]: Becoming Leader. State: Replica: 0e41abe35d8f40a2a01953eea1c5d092, State: Running, Role: LEADER
I20240502 14:25:24.344532 15662 consensus_queue.cc:237] T 4987edf6ce4d44d18162ff06e3f74e23 P 0e41abe35d8f40a2a01953eea1c5d092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 523, Committed index: 523, Last appended: 1.523, Last appended by leader: 523, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } }
I20240502 14:25:24.350179 11788 catalog_manager.cc:5484] T 4987edf6ce4d44d18162ff06e3f74e23 P 0e41abe35d8f40a2a01953eea1c5d092 reported cstate change: term changed from 1 to 2, leader changed from 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66) to 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65). New cstate: current_term: 2 leader_uuid: "0e41abe35d8f40a2a01953eea1c5d092" committed_config { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } } }
W20240502 14:25:24.369119 12022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32810: Illegal state: replica 36e66c7b1a03458b89a91c56a57df72a is not leader of this config: current role FOLLOWER
I20240502 14:25:24.391211 13251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4987edf6ce4d44d18162ff06e3f74e23" candidate_uuid: "0e41abe35d8f40a2a01953eea1c5d092" candidate_term: 2 candidate_status { last_received { term: 1 index: 523 } } ignore_live_leader: true dest_uuid: "93c82e87d8af478292ffe1a3d3986049"
I20240502 14:25:24.391636 13251 raft_consensus.cc:3037] T 4987edf6ce4d44d18162ff06e3f74e23 P 93c82e87d8af478292ffe1a3d3986049 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:25:24.395692 13251 raft_consensus.cc:2454] T 4987edf6ce4d44d18162ff06e3f74e23 P 93c82e87d8af478292ffe1a3d3986049 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e41abe35d8f40a2a01953eea1c5d092 in term 2.
I20240502 14:25:24.469211 12068 raft_consensus.cc:1261] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a [term 2 FOLLOWER]: Refusing update from remote peer 0e41abe35d8f40a2a01953eea1c5d092: Log matching property violated. Preceding OpId in replica: term: 1 index: 523. Preceding OpId from leader: term: 2 index: 525. (index mismatch)
I20240502 14:25:24.469267 13251 raft_consensus.cc:1261] T 4987edf6ce4d44d18162ff06e3f74e23 P 93c82e87d8af478292ffe1a3d3986049 [term 2 FOLLOWER]: Refusing update from remote peer 0e41abe35d8f40a2a01953eea1c5d092: Log matching property violated. Preceding OpId in replica: term: 1 index: 523. Preceding OpId from leader: term: 2 index: 525. (index mismatch)
I20240502 14:25:24.469892 15659 consensus_queue.cc:1035] T 4987edf6ce4d44d18162ff06e3f74e23 P 0e41abe35d8f40a2a01953eea1c5d092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 524, Last known committed idx: 523, Time since last communication: 0.000s
I20240502 14:25:24.470297 15659 consensus_queue.cc:1035] T 4987edf6ce4d44d18162ff06e3f74e23 P 0e41abe35d8f40a2a01953eea1c5d092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 524, Last known committed idx: 523, Time since last communication: 0.000s
W20240502 14:25:24.628922 12134 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.71:33799, user_credentials={real_user=slave}} blocked reactor thread for 57766.4us
W20240502 14:25:24.628937 12135 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.68:37273, user_credentials={real_user=slave}} blocked reactor thread for 52698.4us
I20240502 14:25:24.977768 12045 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:25:25.071408 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:25.095851 11916 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:25:25.200546 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:25.286636 13637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:25.316231 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:25.530376 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:26.107770 11935 consensus_queue.cc:237] T 4987edf6ce4d44d18162ff06e3f74e23 P 0e41abe35d8f40a2a01953eea1c5d092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 550, Committed index: 550, Last appended: 2.550, Last appended by leader: 523, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 551 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } }
I20240502 14:25:26.109691 13249 raft_consensus.cc:1261] T 4987edf6ce4d44d18162ff06e3f74e23 P 93c82e87d8af478292ffe1a3d3986049 [term 2 FOLLOWER]: Refusing update from remote peer 0e41abe35d8f40a2a01953eea1c5d092: Log matching property violated. Preceding OpId in replica: term: 2 index: 550. Preceding OpId from leader: term: 2 index: 551. (index mismatch)
I20240502 14:25:26.110694 15743 consensus_queue.cc:1035] T 4987edf6ce4d44d18162ff06e3f74e23 P 0e41abe35d8f40a2a01953eea1c5d092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 551, Last known committed idx: 550, Time since last communication: 0.000s
I20240502 14:25:26.118379 15456 raft_consensus.cc:2932] T 4987edf6ce4d44d18162ff06e3f74e23 P 0e41abe35d8f40a2a01953eea1c5d092 [term 2 LEADER]: Committing config change with OpId 2.551: config changed from index 88 to 551, VOTER 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66) evicted. New config: { opid_index: 551 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } } }
I20240502 14:25:26.122246 13249 raft_consensus.cc:2932] T 4987edf6ce4d44d18162ff06e3f74e23 P 93c82e87d8af478292ffe1a3d3986049 [term 2 FOLLOWER]: Committing config change with OpId 2.551: config changed from index 88 to 551, VOTER 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66) evicted. New config: { opid_index: 551 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } } }
I20240502 14:25:26.133846 11784 catalog_manager.cc:5484] T 4987edf6ce4d44d18162ff06e3f74e23 P 0e41abe35d8f40a2a01953eea1c5d092 reported cstate change: config changed from index 88 to 551, VOTER 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66) evicted. New cstate: current_term: 2 leader_uuid: "0e41abe35d8f40a2a01953eea1c5d092" committed_config { opid_index: 551 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } } }
I20240502 14:25:26.205932 12045 tablet_service.cc:1514] Processing DeleteTablet for tablet 4987edf6ce4d44d18162ff06e3f74e23 with delete_type TABLET_DATA_TOMBSTONED (TS 36e66c7b1a03458b89a91c56a57df72a not found in new config with opid_index 551) from {username='slave'} at 127.0.0.1:36164
I20240502 14:25:26.273809 15779 tablet_replica.cc:330] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a: stopping tablet replica
I20240502 14:25:26.274783 15779 raft_consensus.cc:2229] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:25:26.275394 15779 raft_consensus.cc:2258] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:25:26.277140 15779 ts_tablet_manager.cc:1875] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:25:26.313119 15779 ts_tablet_manager.cc:1888] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.550
I20240502 14:25:26.313381 15779 log.cc:1196] T 4987edf6ce4d44d18162ff06e3f74e23 P 36e66c7b1a03458b89a91c56a57df72a: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/wals/4987edf6ce4d44d18162ff06e3f74e23
I20240502 14:25:26.314579 11780 catalog_manager.cc:4830] TS 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445): tablet 4987edf6ce4d44d18162ff06e3f74e23 (table rebalance_test_table_0 [id=1ebc3641cd6e480c85c60dd430ccb365]) successfully deleted
I20240502 14:25:27.362882 11916 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:25:27.378506 12045 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:25:27.399005 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:27.403316 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:25:27.445730 12134 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.71:33799, user_credentials={real_user=slave}} blocked reactor thread for 47404.8us
I20240502 14:25:27.529134 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:27.544337 13637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:27.606838 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:28.192617 12068 consensus_queue.cc:237] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 830, Committed index: 830, Last appended: 1.830, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 831 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:25:28.194943 13521 raft_consensus.cc:1261] T 2f0a86c3cb6c48258f4d2f916b192c9c P 375c6d5bdf924866b69b0e6667732692 [term 1 FOLLOWER]: Refusing update from remote peer 36e66c7b1a03458b89a91c56a57df72a: Log matching property violated. Preceding OpId in replica: term: 1 index: 830. Preceding OpId from leader: term: 1 index: 831. (index mismatch)
I20240502 14:25:28.195865 15606 consensus_queue.cc:1035] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 831, Last known committed idx: 830, Time since last communication: 0.000s
I20240502 14:25:28.198372 15550 raft_consensus.cc:2932] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Committing config change with OpId 1.831: config changed from index 156 to 831, VOTER 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65) evicted. New config: { opid_index: 831 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } } }
I20240502 14:25:28.202025 13521 raft_consensus.cc:2932] T 2f0a86c3cb6c48258f4d2f916b192c9c P 375c6d5bdf924866b69b0e6667732692 [term 1 FOLLOWER]: Committing config change with OpId 1.831: config changed from index 156 to 831, VOTER 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65) evicted. New config: { opid_index: 831 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } } }
I20240502 14:25:28.207610 11784 catalog_manager.cc:5484] T 2f0a86c3cb6c48258f4d2f916b192c9c P 36e66c7b1a03458b89a91c56a57df72a reported cstate change: config changed from index 156 to 831, VOTER 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65) evicted. New cstate: current_term: 1 leader_uuid: "36e66c7b1a03458b89a91c56a57df72a" committed_config { opid_index: 831 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } } }
I20240502 14:25:28.361662 11916 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f0a86c3cb6c48258f4d2f916b192c9c with delete_type TABLET_DATA_TOMBSTONED (TS 0e41abe35d8f40a2a01953eea1c5d092 not found in new config with opid_index 831) from {username='slave'} at 127.0.0.1:42830
I20240502 14:25:28.366096 15878 tablet_replica.cc:330] T 2f0a86c3cb6c48258f4d2f916b192c9c P 0e41abe35d8f40a2a01953eea1c5d092: stopping tablet replica
I20240502 14:25:28.366313 15878 raft_consensus.cc:2229] T 2f0a86c3cb6c48258f4d2f916b192c9c P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:25:28.366648 15878 raft_consensus.cc:2258] T 2f0a86c3cb6c48258f4d2f916b192c9c P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:25:28.368269 15878 ts_tablet_manager.cc:1875] T 2f0a86c3cb6c48258f4d2f916b192c9c P 0e41abe35d8f40a2a01953eea1c5d092: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:25:28.394407 15878 ts_tablet_manager.cc:1888] T 2f0a86c3cb6c48258f4d2f916b192c9c P 0e41abe35d8f40a2a01953eea1c5d092: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.830
I20240502 14:25:28.394937 15878 log.cc:1196] T 2f0a86c3cb6c48258f4d2f916b192c9c P 0e41abe35d8f40a2a01953eea1c5d092: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal/wals/2f0a86c3cb6c48258f4d2f916b192c9c
I20240502 14:25:28.396293 11778 catalog_manager.cc:4830] TS 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871): tablet 2f0a86c3cb6c48258f4d2f916b192c9c (table rebalance_test_table_3 [id=423ad8739bd34e329c45d60dce8f9687]) successfully deleted
I20240502 14:25:28.849251 11916 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:25:28.908026 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:28.912801 13637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:28.943172 12045 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:25:28.947137 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:28.979621 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:29.024291 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:29.714923 12195 consensus_queue.cc:237] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 762, Committed index: 762, Last appended: 1.763, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 764 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } }
I20240502 14:25:29.716950 13251 raft_consensus.cc:1261] T 9069116816104c79819b25838836575b P 93c82e87d8af478292ffe1a3d3986049 [term 1 FOLLOWER]: Refusing update from remote peer d9748c575342401bb80683c7e2c4e105: Log matching property violated. Preceding OpId in replica: term: 1 index: 763. Preceding OpId from leader: term: 1 index: 764. (index mismatch)
I20240502 14:25:29.718812 15841 consensus_queue.cc:1035] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 764, Last known committed idx: 762, Time since last communication: 0.000s
I20240502 14:25:29.721158 15507 raft_consensus.cc:2932] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105 [term 1 LEADER]: Committing config change with OpId 1.764: config changed from index 85 to 764, VOTER 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65) evicted. New config: { opid_index: 764 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } } }
I20240502 14:25:29.721841 13251 raft_consensus.cc:2932] T 9069116816104c79819b25838836575b P 93c82e87d8af478292ffe1a3d3986049 [term 1 FOLLOWER]: Committing config change with OpId 1.764: config changed from index 85 to 764, VOTER 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65) evicted. New config: { opid_index: 764 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } } }
I20240502 14:25:29.727607 11784 catalog_manager.cc:5484] T 9069116816104c79819b25838836575b P d9748c575342401bb80683c7e2c4e105 reported cstate change: config changed from index 85 to 764, VOTER 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65) evicted. New cstate: current_term: 1 leader_uuid: "d9748c575342401bb80683c7e2c4e105" committed_config { opid_index: 764 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } } }
I20240502 14:25:29.793349 11916 tablet_service.cc:1514] Processing DeleteTablet for tablet 9069116816104c79819b25838836575b with delete_type TABLET_DATA_TOMBSTONED (TS 0e41abe35d8f40a2a01953eea1c5d092 not found in new config with opid_index 764) from {username='slave'} at 127.0.0.1:42830
I20240502 14:25:29.793958 15956 tablet_replica.cc:330] T 9069116816104c79819b25838836575b P 0e41abe35d8f40a2a01953eea1c5d092: stopping tablet replica
I20240502 14:25:29.794215 15956 raft_consensus.cc:2229] T 9069116816104c79819b25838836575b P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:25:29.794692 15956 pending_rounds.cc:62] T 9069116816104c79819b25838836575b P 0e41abe35d8f40a2a01953eea1c5d092: Trying to abort 1 pending ops.
I20240502 14:25:29.794857 15956 pending_rounds.cc:68] T 9069116816104c79819b25838836575b P 0e41abe35d8f40a2a01953eea1c5d092: Aborting op as it isn't in flight: id { term: 1 index: 763 } timestamp: 7023247072106774528 op_type: WRITE_OP write_request { tablet_id: "9069116816104c79819b25838836575b" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\373\227(w\007s$\253\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""7e73516289c87393" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023247072049754112 authz_token { token_data: "\010\330\306\316\261\006\"3\n\005slave\022*\n fe50532b3c3d46b7bdae21e985f105f6\020\001\030\001 \001(\001" signature: "<redacted>""ns\201\356p\0067gq\215\371\256O\353\177I!\257\265\336=\215y\022\027\265cv\247\027L\277\254\253\351:\216\266\222\365q\346\256\364xr\006\024\330\3735\036\tq\235\037\005y\224\232\260n\027\342" signing_key_seq_num: 0 } } request_id { client_id: "315f81c425d54bc2b2266fadb7083af2" seq_no: 2226 first_incomplete_seq_no: 2226 attempt_no: 0 }
I20240502 14:25:29.814713 15956 raft_consensus.cc:2258] T 9069116816104c79819b25838836575b P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:25:29.815991 15956 ts_tablet_manager.cc:1875] T 9069116816104c79819b25838836575b P 0e41abe35d8f40a2a01953eea1c5d092: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:25:29.822661 15956 ts_tablet_manager.cc:1888] T 9069116816104c79819b25838836575b P 0e41abe35d8f40a2a01953eea1c5d092: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.763
I20240502 14:25:29.824383 15956 log.cc:1196] T 9069116816104c79819b25838836575b P 0e41abe35d8f40a2a01953eea1c5d092: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal/wals/9069116816104c79819b25838836575b
I20240502 14:25:29.826411 11778 catalog_manager.cc:4830] TS 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871): tablet 9069116816104c79819b25838836575b (table rebalance_test_table_1 [id=fe50532b3c3d46b7bdae21e985f105f6]) successfully deleted
I20240502 14:25:30.600153 11916 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:25:30.629695 12045 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:25:30.815903 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:30.874399 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:30.952670 13637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:31.049515 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:31.073835 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:31.490779 12068 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "439f0129f4bd49b7b582be22be83e0b4"
dest_uuid: "36e66c7b1a03458b89a91c56a57df72a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49102
I20240502 14:25:31.490958 12068 raft_consensus.cc:595] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Received request to transfer leadership
W20240502 14:25:31.506954 12027 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:31.523003 12026 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:31.544054 12027 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:31.561005 12026 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:31.576736 12027 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:31.586289 12027 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:31.596315 12027 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:31.614804 12027 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:31.648288 12027 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:31.663344 12027 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:31.676295 12027 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:31.691509 12027 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:31.705204 12027 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:31.728250 12027 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:31.759810 12027 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:31.782176 12027 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:31.802320 12027 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
W20240502 14:25:31.822057 12026 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Service unavailable: leader transfer in progress
I20240502 14:25:31.825559 15937 raft_consensus.cc:979] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a: : Instructing follower 0e41abe35d8f40a2a01953eea1c5d092 to start an election
I20240502 14:25:31.825691 15937 raft_consensus.cc:1067] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Signalling peer 0e41abe35d8f40a2a01953eea1c5d092 to start an election
I20240502 14:25:31.826354 11931 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "439f0129f4bd49b7b582be22be83e0b4"
dest_uuid: "0e41abe35d8f40a2a01953eea1c5d092"
from {username='slave'} at 127.5.107.66:60283
I20240502 14:25:31.826529 11931 raft_consensus.cc:482] T 439f0129f4bd49b7b582be22be83e0b4 P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:25:31.826617 11931 raft_consensus.cc:3037] T 439f0129f4bd49b7b582be22be83e0b4 P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:25:31.829392 11931 raft_consensus.cc:504] T 439f0129f4bd49b7b582be22be83e0b4 P 0e41abe35d8f40a2a01953eea1c5d092 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } }
I20240502 14:25:31.831887 12068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "439f0129f4bd49b7b582be22be83e0b4" candidate_uuid: "0e41abe35d8f40a2a01953eea1c5d092" candidate_term: 2 candidate_status { last_received { term: 1 index: 1011 } } ignore_live_leader: true dest_uuid: "36e66c7b1a03458b89a91c56a57df72a"
I20240502 14:25:31.832065 12068 raft_consensus.cc:3032] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:25:31.831995 11931 leader_election.cc:288] T 439f0129f4bd49b7b582be22be83e0b4 P 0e41abe35d8f40a2a01953eea1c5d092 [CANDIDATE]: Term 2 election: Requested vote from peers 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445), 723fb53f34c94c6aa5f7ee3a65711268 (127.5.107.69:33193)
I20240502 14:25:31.832134 12068 raft_consensus.cc:729] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 36e66c7b1a03458b89a91c56a57df72a, State: Running, Role: LEADER
I20240502 14:25:31.832357 12068 consensus_queue.cc:260] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1011, Committed index: 1011, Last appended: 1.1011, Last appended by leader: 1011, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } }
I20240502 14:25:31.832506 13384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "439f0129f4bd49b7b582be22be83e0b4" candidate_uuid: "0e41abe35d8f40a2a01953eea1c5d092" candidate_term: 2 candidate_status { last_received { term: 1 index: 1011 } } ignore_live_leader: true dest_uuid: "723fb53f34c94c6aa5f7ee3a65711268"
I20240502 14:25:31.832669 12068 raft_consensus.cc:3037] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:25:31.832891 13384 raft_consensus.cc:3037] T 439f0129f4bd49b7b582be22be83e0b4 P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:25:31.835072 13384 raft_consensus.cc:2454] T 439f0129f4bd49b7b582be22be83e0b4 P 723fb53f34c94c6aa5f7ee3a65711268 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e41abe35d8f40a2a01953eea1c5d092 in term 2.
I20240502 14:25:31.835173 12068 raft_consensus.cc:2454] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e41abe35d8f40a2a01953eea1c5d092 in term 2.
I20240502 14:25:31.835497 11868 leader_election.cc:302] T 439f0129f4bd49b7b582be22be83e0b4 P 0e41abe35d8f40a2a01953eea1c5d092 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e41abe35d8f40a2a01953eea1c5d092, 723fb53f34c94c6aa5f7ee3a65711268; no voters:
I20240502 14:25:31.835780 15924 raft_consensus.cc:2781] T 439f0129f4bd49b7b582be22be83e0b4 P 0e41abe35d8f40a2a01953eea1c5d092 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:25:31.835922 15924 raft_consensus.cc:686] T 439f0129f4bd49b7b582be22be83e0b4 P 0e41abe35d8f40a2a01953eea1c5d092 [term 2 LEADER]: Becoming Leader. State: Replica: 0e41abe35d8f40a2a01953eea1c5d092, State: Running, Role: LEADER
I20240502 14:25:31.836179 15924 consensus_queue.cc:237] T 439f0129f4bd49b7b582be22be83e0b4 P 0e41abe35d8f40a2a01953eea1c5d092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1011, Committed index: 1011, Last appended: 1.1011, Last appended by leader: 1011, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } }
I20240502 14:25:31.842083 11784 catalog_manager.cc:5484] T 439f0129f4bd49b7b582be22be83e0b4 P 0e41abe35d8f40a2a01953eea1c5d092 reported cstate change: term changed from 1 to 2, leader changed from 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66) to 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65). New cstate: current_term: 2 leader_uuid: "0e41abe35d8f40a2a01953eea1c5d092" committed_config { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } }
W20240502 14:25:31.842768 12026 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32822: Illegal state: replica 36e66c7b1a03458b89a91c56a57df72a is not leader of this config: current role FOLLOWER
I20240502 14:25:31.920848 11916 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:31.934811 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:31.941892 12045 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:25:31.964308 13386 raft_consensus.cc:1261] T 439f0129f4bd49b7b582be22be83e0b4 P 723fb53f34c94c6aa5f7ee3a65711268 [term 2 FOLLOWER]: Refusing update from remote peer 0e41abe35d8f40a2a01953eea1c5d092: Log matching property violated. Preceding OpId in replica: term: 1 index: 1011. Preceding OpId from leader: term: 2 index: 1013. (index mismatch)
I20240502 14:25:31.965014 15924 consensus_queue.cc:1035] T 439f0129f4bd49b7b582be22be83e0b4 P 0e41abe35d8f40a2a01953eea1c5d092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1012, Last known committed idx: 1011, Time since last communication: 0.000s
I20240502 14:25:31.966598 12068 raft_consensus.cc:1261] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a [term 2 FOLLOWER]: Refusing update from remote peer 0e41abe35d8f40a2a01953eea1c5d092: Log matching property violated. Preceding OpId in replica: term: 1 index: 1011. Preceding OpId from leader: term: 2 index: 1013. (index mismatch)
I20240502 14:25:31.967100 16030 consensus_queue.cc:1035] T 439f0129f4bd49b7b582be22be83e0b4 P 0e41abe35d8f40a2a01953eea1c5d092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1012, Last known committed idx: 1011, Time since last communication: 0.000s
I20240502 14:25:31.993871 16000 mvcc.cc:205] Tried to move back new op lower bound from 7023247081321213952 to 7023247080803459072. Current Snapshot: MvccSnapshot[applied={T|T < 7023247081321213952}]
I20240502 14:25:32.303987 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:32.307190 13637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:32.361838 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:32.480495 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 14:25:32.525646 11999 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.69:33193, user_credentials={real_user=slave}} blocked reactor thread for 54062.4us
I20240502 14:25:33.657402 11916 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:33.667766 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:33.671869 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:33.689606 12045 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:25:33.695672 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:33.707741 13637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:33.794560 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:34.215687 12068 consensus_queue.cc:237] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1055, Committed index: 1055, Last appended: 1.1055, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1056 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:25:34.217367 13521 raft_consensus.cc:1261] T 003642773a724534847246de230ea34b P 375c6d5bdf924866b69b0e6667732692 [term 1 FOLLOWER]: Refusing update from remote peer 36e66c7b1a03458b89a91c56a57df72a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1055. Preceding OpId from leader: term: 1 index: 1056. (index mismatch)
I20240502 14:25:34.218112 16089 consensus_queue.cc:1035] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1056, Last known committed idx: 1055, Time since last communication: 0.000s
I20240502 14:25:34.220644 16089 raft_consensus.cc:2932] T 003642773a724534847246de230ea34b P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Committing config change with OpId 1.1056: config changed from index 81 to 1056, VOTER d9748c575342401bb80683c7e2c4e105 (127.5.107.67) evicted. New config: { opid_index: 1056 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } } }
I20240502 14:25:34.221093 13521 raft_consensus.cc:2932] T 003642773a724534847246de230ea34b P 375c6d5bdf924866b69b0e6667732692 [term 1 FOLLOWER]: Committing config change with OpId 1.1056: config changed from index 81 to 1056, VOTER d9748c575342401bb80683c7e2c4e105 (127.5.107.67) evicted. New config: { opid_index: 1056 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } } }
I20240502 14:25:34.226078 11784 catalog_manager.cc:5484] T 003642773a724534847246de230ea34b P 375c6d5bdf924866b69b0e6667732692 reported cstate change: config changed from index 81 to 1056, VOTER d9748c575342401bb80683c7e2c4e105 (127.5.107.67) evicted. New cstate: current_term: 1 leader_uuid: "36e66c7b1a03458b89a91c56a57df72a" committed_config { opid_index: 1056 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } } }
I20240502 14:25:34.309379 12178 tablet_service.cc:1514] Processing DeleteTablet for tablet 003642773a724534847246de230ea34b with delete_type TABLET_DATA_TOMBSTONED (TS d9748c575342401bb80683c7e2c4e105 not found in new config with opid_index 1056) from {username='slave'} at 127.0.0.1:45972
I20240502 14:25:34.395205 16184 tablet_replica.cc:330] T 003642773a724534847246de230ea34b P d9748c575342401bb80683c7e2c4e105: stopping tablet replica
I20240502 14:25:34.395488 16184 raft_consensus.cc:2229] T 003642773a724534847246de230ea34b P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:25:34.398684 16184 raft_consensus.cc:2258] T 003642773a724534847246de230ea34b P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:25:34.406539 16184 ts_tablet_manager.cc:1875] T 003642773a724534847246de230ea34b P d9748c575342401bb80683c7e2c4e105: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:25:34.419512 16184 ts_tablet_manager.cc:1888] T 003642773a724534847246de230ea34b P d9748c575342401bb80683c7e2c4e105: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1055
I20240502 14:25:34.444751 16184 log.cc:1196] T 003642773a724534847246de230ea34b P d9748c575342401bb80683c7e2c4e105: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal/wals/003642773a724534847246de230ea34b
I20240502 14:25:34.445879 11777 catalog_manager.cc:4830] TS d9748c575342401bb80683c7e2c4e105 (127.5.107.67:45923): tablet 003642773a724534847246de230ea34b (table rebalance_test_table_2 [id=ec56bc8cfbc04f26b92384c442eac5b8]) successfully deleted
I20240502 14:25:35.172322 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:35.173885 11916 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:35.192513 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:35.271575 13637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:35.360880 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:35.369998 12045 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:25:35.391484 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:35.606891 12068 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d55e54737408455fadf1a1b4041edd0a"
dest_uuid: "36e66c7b1a03458b89a91c56a57df72a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60682
I20240502 14:25:35.607117 12068 raft_consensus.cc:595] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Received request to transfer leadership
I20240502 14:25:35.610141 16213 raft_consensus.cc:979] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a: : Instructing follower b0d1c78721a746e99d0d3056f1e31371 to start an election
I20240502 14:25:35.610364 16213 raft_consensus.cc:1067] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Signalling peer b0d1c78721a746e99d0d3056f1e31371 to start an election
I20240502 14:25:35.611513 13654 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d55e54737408455fadf1a1b4041edd0a"
dest_uuid: "b0d1c78721a746e99d0d3056f1e31371"
from {username='slave'} at 127.5.107.66:50745
I20240502 14:25:35.611828 13654 raft_consensus.cc:482] T d55e54737408455fadf1a1b4041edd0a P b0d1c78721a746e99d0d3056f1e31371 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:25:35.611985 13654 raft_consensus.cc:3037] T d55e54737408455fadf1a1b4041edd0a P b0d1c78721a746e99d0d3056f1e31371 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:25:35.616160 13654 raft_consensus.cc:504] T d55e54737408455fadf1a1b4041edd0a P b0d1c78721a746e99d0d3056f1e31371 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } }
I20240502 14:25:35.619163 13654 leader_election.cc:288] T d55e54737408455fadf1a1b4041edd0a P b0d1c78721a746e99d0d3056f1e31371 [CANDIDATE]: Term 2 election: Requested vote from peers 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445), d9748c575342401bb80683c7e2c4e105 (127.5.107.67:45923)
I20240502 14:25:35.620311 12068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55e54737408455fadf1a1b4041edd0a" candidate_uuid: "b0d1c78721a746e99d0d3056f1e31371" candidate_term: 2 candidate_status { last_received { term: 1 index: 734 } } ignore_live_leader: true dest_uuid: "36e66c7b1a03458b89a91c56a57df72a"
I20240502 14:25:35.620514 12068 raft_consensus.cc:3032] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:25:35.620657 12068 raft_consensus.cc:729] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 36e66c7b1a03458b89a91c56a57df72a, State: Running, Role: LEADER
I20240502 14:25:35.620875 12200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55e54737408455fadf1a1b4041edd0a" candidate_uuid: "b0d1c78721a746e99d0d3056f1e31371" candidate_term: 2 candidate_status { last_received { term: 1 index: 734 } } ignore_live_leader: true dest_uuid: "d9748c575342401bb80683c7e2c4e105"
I20240502 14:25:35.620927 12068 consensus_queue.cc:260] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 734, Committed index: 734, Last appended: 1.734, Last appended by leader: 734, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } }
I20240502 14:25:35.621050 12200 raft_consensus.cc:3037] T d55e54737408455fadf1a1b4041edd0a P d9748c575342401bb80683c7e2c4e105 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:25:35.621490 12068 raft_consensus.cc:3037] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:25:35.623628 12200 raft_consensus.cc:2454] T d55e54737408455fadf1a1b4041edd0a P d9748c575342401bb80683c7e2c4e105 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0d1c78721a746e99d0d3056f1e31371 in term 2.
I20240502 14:25:35.623994 12068 raft_consensus.cc:2454] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0d1c78721a746e99d0d3056f1e31371 in term 2.
I20240502 14:25:35.624089 13589 leader_election.cc:302] T d55e54737408455fadf1a1b4041edd0a P b0d1c78721a746e99d0d3056f1e31371 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0d1c78721a746e99d0d3056f1e31371, d9748c575342401bb80683c7e2c4e105; no voters:
I20240502 14:25:35.640136 12200 raft_consensus.cc:1226] T d55e54737408455fadf1a1b4041edd0a P d9748c575342401bb80683c7e2c4e105 [term 2 FOLLOWER]: Rejecting Update request from peer 36e66c7b1a03458b89a91c56a57df72a for earlier term 1. Current term is 2. Ops: []
I20240502 14:25:35.655629 16254 raft_consensus.cc:2781] T d55e54737408455fadf1a1b4041edd0a P b0d1c78721a746e99d0d3056f1e31371 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:25:35.674980 16254 raft_consensus.cc:686] T d55e54737408455fadf1a1b4041edd0a P b0d1c78721a746e99d0d3056f1e31371 [term 2 LEADER]: Becoming Leader. State: Replica: b0d1c78721a746e99d0d3056f1e31371, State: Running, Role: LEADER
I20240502 14:25:35.676528 16254 consensus_queue.cc:237] T d55e54737408455fadf1a1b4041edd0a P b0d1c78721a746e99d0d3056f1e31371 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 734, Committed index: 734, Last appended: 1.734, Last appended by leader: 734, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } }
I20240502 14:25:35.691852 11784 catalog_manager.cc:5484] T d55e54737408455fadf1a1b4041edd0a P b0d1c78721a746e99d0d3056f1e31371 reported cstate change: term changed from 1 to 2, leader changed from 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66) to b0d1c78721a746e99d0d3056f1e31371 (127.5.107.71). New cstate: current_term: 2 leader_uuid: "b0d1c78721a746e99d0d3056f1e31371" committed_config { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } }
W20240502 14:25:35.707561 12026 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32836: Illegal state: replica 36e66c7b1a03458b89a91c56a57df72a is not leader of this config: current role FOLLOWER
W20240502 14:25:35.748901 12157 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40002: Illegal state: replica d9748c575342401bb80683c7e2c4e105 is not leader of this config: current role FOLLOWER
I20240502 14:25:35.772305 12200 raft_consensus.cc:1261] T d55e54737408455fadf1a1b4041edd0a P d9748c575342401bb80683c7e2c4e105 [term 2 FOLLOWER]: Refusing update from remote peer b0d1c78721a746e99d0d3056f1e31371: Log matching property violated. Preceding OpId in replica: term: 1 index: 734. Preceding OpId from leader: term: 2 index: 736. (index mismatch)
I20240502 14:25:35.772936 12068 raft_consensus.cc:1261] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a [term 2 FOLLOWER]: Refusing update from remote peer b0d1c78721a746e99d0d3056f1e31371: Log matching property violated. Preceding OpId in replica: term: 1 index: 734. Preceding OpId from leader: term: 2 index: 736. (index mismatch)
I20240502 14:25:35.772986 16258 consensus_queue.cc:1035] T d55e54737408455fadf1a1b4041edd0a P b0d1c78721a746e99d0d3056f1e31371 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 735, Last known committed idx: 733, Time since last communication: 0.000s
I20240502 14:25:35.774181 16258 consensus_queue.cc:1035] T d55e54737408455fadf1a1b4041edd0a P b0d1c78721a746e99d0d3056f1e31371 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 735, Last known committed idx: 734, Time since last communication: 0.000s
I20240502 14:25:36.114634 11916 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:36.169548 12045 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:25:36.279894 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:36.320118 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:36.321600 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:36.376447 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:36.470294 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:36.963596 13655 consensus_queue.cc:237] T d55e54737408455fadf1a1b4041edd0a P b0d1c78721a746e99d0d3056f1e31371 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 773, Committed index: 773, Last appended: 2.773, Last appended by leader: 734, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 774 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } }
I20240502 14:25:36.964953 12195 raft_consensus.cc:1261] T d55e54737408455fadf1a1b4041edd0a P d9748c575342401bb80683c7e2c4e105 [term 2 FOLLOWER]: Refusing update from remote peer b0d1c78721a746e99d0d3056f1e31371: Log matching property violated. Preceding OpId in replica: term: 2 index: 773. Preceding OpId from leader: term: 2 index: 774. (index mismatch)
I20240502 14:25:36.965521 16254 consensus_queue.cc:1035] T d55e54737408455fadf1a1b4041edd0a P b0d1c78721a746e99d0d3056f1e31371 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 774, Last known committed idx: 773, Time since last communication: 0.000s
I20240502 14:25:36.967811 16254 raft_consensus.cc:2932] T d55e54737408455fadf1a1b4041edd0a P b0d1c78721a746e99d0d3056f1e31371 [term 2 LEADER]: Committing config change with OpId 2.774: config changed from index 71 to 774, VOTER 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66) evicted. New config: { opid_index: 774 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } }
I20240502 14:25:36.970944 12195 raft_consensus.cc:2932] T d55e54737408455fadf1a1b4041edd0a P d9748c575342401bb80683c7e2c4e105 [term 2 FOLLOWER]: Committing config change with OpId 2.774: config changed from index 71 to 774, VOTER 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66) evicted. New config: { opid_index: 774 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } }
I20240502 14:25:36.980314 11784 catalog_manager.cc:5484] T d55e54737408455fadf1a1b4041edd0a P d9748c575342401bb80683c7e2c4e105 reported cstate change: config changed from index 71 to 774, VOTER 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66) evicted. New cstate: current_term: 2 leader_uuid: "b0d1c78721a746e99d0d3056f1e31371" committed_config { opid_index: 774 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } }
I20240502 14:25:37.041445 12045 tablet_service.cc:1514] Processing DeleteTablet for tablet d55e54737408455fadf1a1b4041edd0a with delete_type TABLET_DATA_TOMBSTONED (TS 36e66c7b1a03458b89a91c56a57df72a not found in new config with opid_index 774) from {username='slave'} at 127.0.0.1:36164
I20240502 14:25:37.042050 16341 tablet_replica.cc:330] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a: stopping tablet replica
I20240502 14:25:37.042456 16341 raft_consensus.cc:2229] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:25:37.043093 16341 raft_consensus.cc:2258] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:25:37.044049 16341 ts_tablet_manager.cc:1875] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:25:37.051056 16341 ts_tablet_manager.cc:1888] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.773
I20240502 14:25:37.051247 16341 log.cc:1196] T d55e54737408455fadf1a1b4041edd0a P 36e66c7b1a03458b89a91c56a57df72a: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/wals/d55e54737408455fadf1a1b4041edd0a
I20240502 14:25:37.059633 11780 catalog_manager.cc:4830] TS 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445): tablet d55e54737408455fadf1a1b4041edd0a (table rebalance_test_table_4 [id=cc8c51baff5f4081aca1cf7908b85057]) successfully deleted
I20240502 14:25:38.830197 11916 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:38.868721 12045 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:25:38.910521 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:38.918509 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:38.925621 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:38.939215 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:38.947093 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:39.871167 11935 consensus_queue.cc:237] T 439f0129f4bd49b7b582be22be83e0b4 P 0e41abe35d8f40a2a01953eea1c5d092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1178, Committed index: 1178, Last appended: 2.1178, Last appended by leader: 1011, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1179 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } }
I20240502 14:25:39.877408 13384 raft_consensus.cc:1261] T 439f0129f4bd49b7b582be22be83e0b4 P 723fb53f34c94c6aa5f7ee3a65711268 [term 2 FOLLOWER]: Refusing update from remote peer 0e41abe35d8f40a2a01953eea1c5d092: Log matching property violated. Preceding OpId in replica: term: 2 index: 1178. Preceding OpId from leader: term: 2 index: 1179. (index mismatch)
I20240502 14:25:39.879890 16107 consensus_queue.cc:1035] T 439f0129f4bd49b7b582be22be83e0b4 P 0e41abe35d8f40a2a01953eea1c5d092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1179, Last known committed idx: 1178, Time since last communication: 0.000s
I20240502 14:25:39.883174 16107 raft_consensus.cc:2932] T 439f0129f4bd49b7b582be22be83e0b4 P 0e41abe35d8f40a2a01953eea1c5d092 [term 2 LEADER]: Committing config change with OpId 2.1179: config changed from index 52 to 1179, VOTER 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66) evicted. New config: { opid_index: 1179 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } }
I20240502 14:25:39.885957 13387 raft_consensus.cc:2932] T 439f0129f4bd49b7b582be22be83e0b4 P 723fb53f34c94c6aa5f7ee3a65711268 [term 2 FOLLOWER]: Committing config change with OpId 2.1179: config changed from index 52 to 1179, VOTER 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66) evicted. New config: { opid_index: 1179 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } }
I20240502 14:25:39.900171 11788 catalog_manager.cc:5484] T 439f0129f4bd49b7b582be22be83e0b4 P 0e41abe35d8f40a2a01953eea1c5d092 reported cstate change: config changed from index 52 to 1179, VOTER 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66) evicted. New cstate: current_term: 2 leader_uuid: "0e41abe35d8f40a2a01953eea1c5d092" committed_config { opid_index: 1179 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } }
I20240502 14:25:40.013222 12045 tablet_service.cc:1514] Processing DeleteTablet for tablet 439f0129f4bd49b7b582be22be83e0b4 with delete_type TABLET_DATA_TOMBSTONED (TS 36e66c7b1a03458b89a91c56a57df72a not found in new config with opid_index 1179) from {username='slave'} at 127.0.0.1:36164
I20240502 14:25:40.046911 16516 tablet_replica.cc:330] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a: stopping tablet replica
I20240502 14:25:40.051363 16516 raft_consensus.cc:2229] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:25:40.051679 16516 raft_consensus.cc:2258] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:25:40.055313 16516 ts_tablet_manager.cc:1875] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:25:40.061532 16516 ts_tablet_manager.cc:1888] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1178
I20240502 14:25:40.061671 16516 log.cc:1196] T 439f0129f4bd49b7b582be22be83e0b4 P 36e66c7b1a03458b89a91c56a57df72a: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/wals/439f0129f4bd49b7b582be22be83e0b4
I20240502 14:25:40.065259 11780 catalog_manager.cc:4830] TS 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445): tablet 439f0129f4bd49b7b582be22be83e0b4 (table rebalance_test_table_2 [id=ec56bc8cfbc04f26b92384c442eac5b8]) successfully deleted
I20240502 14:25:40.659899 12068 consensus_queue.cc:237] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 852, Committed index: 852, Last appended: 1.853, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 854 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } }
W20240502 14:25:40.661865 12002 consensus_peers.cc:485] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a -> Peer 93c82e87d8af478292ffe1a3d3986049 (127.5.107.68:37273): Couldn't send request to peer 93c82e87d8af478292ffe1a3d3986049. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f61f17c159094ff59d935ed0ef6381b8. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:25:40.664426 11935 raft_consensus.cc:1261] T f61f17c159094ff59d935ed0ef6381b8 P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Refusing update from remote peer 36e66c7b1a03458b89a91c56a57df72a: Log matching property violated. Preceding OpId in replica: term: 1 index: 853. Preceding OpId from leader: term: 1 index: 854. (index mismatch)
I20240502 14:25:40.669144 16089 consensus_queue.cc:1035] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 854, Last known committed idx: 853, Time since last communication: 0.000s
I20240502 14:25:40.672883 16506 raft_consensus.cc:2932] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Committing config change with OpId 1.854: config changed from index -1 to 854, VOTER 93c82e87d8af478292ffe1a3d3986049 (127.5.107.68) added. New config: { opid_index: 854 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } } }
I20240502 14:25:40.673529 11935 raft_consensus.cc:2932] T f61f17c159094ff59d935ed0ef6381b8 P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Committing config change with OpId 1.854: config changed from index -1 to 854, VOTER 93c82e87d8af478292ffe1a3d3986049 (127.5.107.68) added. New config: { opid_index: 854 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } } }
I20240502 14:25:40.680487 11784 catalog_manager.cc:5484] T f61f17c159094ff59d935ed0ef6381b8 P 0e41abe35d8f40a2a01953eea1c5d092 reported cstate change: config changed from index -1 to 854, VOTER 93c82e87d8af478292ffe1a3d3986049 (127.5.107.68) added. New cstate: current_term: 1 leader_uuid: "36e66c7b1a03458b89a91c56a57df72a" committed_config { opid_index: 854 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } } }
I20240502 14:25:41.155375 16573 ts_tablet_manager.cc:902] T f61f17c159094ff59d935ed0ef6381b8 P 93c82e87d8af478292ffe1a3d3986049: Initiating tablet copy from peer 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445)
I20240502 14:25:41.239933 16573 tablet_copy_client.cc:308] T f61f17c159094ff59d935ed0ef6381b8 P 93c82e87d8af478292ffe1a3d3986049: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:35445
I20240502 14:25:41.240543 12088 tablet_copy_service.cc:140] P 36e66c7b1a03458b89a91c56a57df72a: Received BeginTabletCopySession request for tablet f61f17c159094ff59d935ed0ef6381b8 from peer 93c82e87d8af478292ffe1a3d3986049 ({username='slave'} at 127.5.107.68:41265)
I20240502 14:25:41.240726 12088 tablet_copy_service.cc:161] P 36e66c7b1a03458b89a91c56a57df72a: Beginning new tablet copy session on tablet f61f17c159094ff59d935ed0ef6381b8 from peer 93c82e87d8af478292ffe1a3d3986049 at {username='slave'} at 127.5.107.68:41265: session id = 93c82e87d8af478292ffe1a3d3986049-f61f17c159094ff59d935ed0ef6381b8
I20240502 14:25:41.243418 12088 tablet_copy_source_session.cc:205] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:25:41.249449 16573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f61f17c159094ff59d935ed0ef6381b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:25:41.269043 16573 tablet_copy_client.cc:783] T f61f17c159094ff59d935ed0ef6381b8 P 93c82e87d8af478292ffe1a3d3986049: tablet copy: Starting download of 0 data blocks...
I20240502 14:25:41.276284 16573 tablet_copy_client.cc:647] T f61f17c159094ff59d935ed0ef6381b8 P 93c82e87d8af478292ffe1a3d3986049: tablet copy: Starting download of 1 WAL segments...
I20240502 14:25:41.281381 16573 tablet_copy_client.cc:523] T f61f17c159094ff59d935ed0ef6381b8 P 93c82e87d8af478292ffe1a3d3986049: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:25:41.285034 16573 tablet_bootstrap.cc:492] T f61f17c159094ff59d935ed0ef6381b8 P 93c82e87d8af478292ffe1a3d3986049: Bootstrap starting.
I20240502 14:25:41.744153 11916 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:41.793596 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:41.814276 12045 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:25:41.885332 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:41.929816 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:42.104015 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:42.163386 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:25:43.437696 12002 consensus_peers.cc:485] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a -> Peer 93c82e87d8af478292ffe1a3d3986049 (127.5.107.68:37273): Couldn't send request to peer 93c82e87d8af478292ffe1a3d3986049. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240502 14:25:43.656138 16573 tablet_bootstrap.cc:492] T f61f17c159094ff59d935ed0ef6381b8 P 93c82e87d8af478292ffe1a3d3986049: Bootstrap replayed 1/1 log segments. Stats: ops{read=866 overwritten=0 applied=866 ignored=0} inserts{seen=1538 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:25:43.658524 16573 tablet_bootstrap.cc:492] T f61f17c159094ff59d935ed0ef6381b8 P 93c82e87d8af478292ffe1a3d3986049: Bootstrap complete.
I20240502 14:25:43.659394 16573 ts_tablet_manager.cc:1367] T f61f17c159094ff59d935ed0ef6381b8 P 93c82e87d8af478292ffe1a3d3986049: Time spent bootstrapping tablet: real 2.374s user 1.034s sys 0.078s
I20240502 14:25:43.660715 16573 raft_consensus.cc:348] T f61f17c159094ff59d935ed0ef6381b8 P 93c82e87d8af478292ffe1a3d3986049 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 854 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } }
I20240502 14:25:43.662007 16573 raft_consensus.cc:729] T f61f17c159094ff59d935ed0ef6381b8 P 93c82e87d8af478292ffe1a3d3986049 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93c82e87d8af478292ffe1a3d3986049, State: Initialized, Role: FOLLOWER
I20240502 14:25:43.662547 16573 consensus_queue.cc:260] T f61f17c159094ff59d935ed0ef6381b8 P 93c82e87d8af478292ffe1a3d3986049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 866, Last appended: 1.866, Last appended by leader: 866, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 854 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } }
I20240502 14:25:43.670872 16573 ts_tablet_manager.cc:1398] T f61f17c159094ff59d935ed0ef6381b8 P 93c82e87d8af478292ffe1a3d3986049: Time spent starting tablet: real 0.011s user 0.001s sys 0.003s
I20240502 14:25:43.678354 12088 tablet_copy_service.cc:342] P 36e66c7b1a03458b89a91c56a57df72a: Request end of tablet copy session 93c82e87d8af478292ffe1a3d3986049-f61f17c159094ff59d935ed0ef6381b8 received from {username='slave'} at 127.5.107.68:41265
I20240502 14:25:43.678517 12088 tablet_copy_service.cc:433] P 36e66c7b1a03458b89a91c56a57df72a: ending tablet copy session 93c82e87d8af478292ffe1a3d3986049-f61f17c159094ff59d935ed0ef6381b8 on tablet f61f17c159094ff59d935ed0ef6381b8 with peer 93c82e87d8af478292ffe1a3d3986049
I20240502 14:25:43.742422 11916 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:43.809300 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:43.844681 12045 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:25:43.913101 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:43.913816 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:43.914386 13251 raft_consensus.cc:1203] T f61f17c159094ff59d935ed0ef6381b8 P 93c82e87d8af478292ffe1a3d3986049 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.853->[1.854-1.897] Dedup: 1.866->[1.867-1.897]
I20240502 14:25:43.943230 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:44.053840 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 14:25:45.237913 13456 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.68:37273, user_credentials={real_user=slave}} blocked reactor thread for 45755.6us
I20240502 14:25:45.293027 11916 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:45.401324 12045 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:25:45.466338 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:45.496512 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:45.534227 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:45.538220 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:45.545152 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:45.945987 12068 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f61f17c159094ff59d935ed0ef6381b8"
dest_uuid: "36e66c7b1a03458b89a91c56a57df72a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57440
I20240502 14:25:45.946231 12068 raft_consensus.cc:595] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Received request to transfer leadership
W20240502 14:25:46.020718 12026 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32836: Service unavailable: leader transfer in progress
W20240502 14:25:46.027993 12026 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32836: Service unavailable: leader transfer in progress
W20240502 14:25:46.033382 12026 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32836: Service unavailable: leader transfer in progress
W20240502 14:25:46.040045 12026 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32836: Service unavailable: leader transfer in progress
W20240502 14:25:46.050942 12022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32836: Service unavailable: leader transfer in progress
W20240502 14:25:46.057790 12022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32836: Service unavailable: leader transfer in progress
W20240502 14:25:46.065903 12022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32836: Service unavailable: leader transfer in progress
W20240502 14:25:46.078449 12026 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32836: Service unavailable: leader transfer in progress
W20240502 14:25:46.090634 12026 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32836: Service unavailable: leader transfer in progress
W20240502 14:25:46.104671 12022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32836: Service unavailable: leader transfer in progress
W20240502 14:25:46.120362 12026 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32836: Service unavailable: leader transfer in progress
W20240502 14:25:46.135128 12022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32836: Service unavailable: leader transfer in progress
W20240502 14:25:46.156113 12026 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32836: Service unavailable: leader transfer in progress
W20240502 14:25:46.178376 12026 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32836: Service unavailable: leader transfer in progress
W20240502 14:25:46.198340 12026 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32836: Service unavailable: leader transfer in progress
W20240502 14:25:46.217268 12026 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32836: Service unavailable: leader transfer in progress
W20240502 14:25:46.238895 12022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32836: Service unavailable: leader transfer in progress
W20240502 14:25:46.263090 12026 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32836: Service unavailable: leader transfer in progress
W20240502 14:25:46.285807 12026 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32836: Service unavailable: leader transfer in progress
I20240502 14:25:46.293694 16714 raft_consensus.cc:979] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a: : Instructing follower 0e41abe35d8f40a2a01953eea1c5d092 to start an election
I20240502 14:25:46.293949 16714 raft_consensus.cc:1067] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Signalling peer 0e41abe35d8f40a2a01953eea1c5d092 to start an election
I20240502 14:25:46.294617 11935 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f61f17c159094ff59d935ed0ef6381b8"
dest_uuid: "0e41abe35d8f40a2a01953eea1c5d092"
from {username='slave'} at 127.5.107.66:60283
I20240502 14:25:46.294819 11935 raft_consensus.cc:482] T f61f17c159094ff59d935ed0ef6381b8 P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:25:46.294910 11935 raft_consensus.cc:3037] T f61f17c159094ff59d935ed0ef6381b8 P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:25:46.298707 11935 raft_consensus.cc:504] T f61f17c159094ff59d935ed0ef6381b8 P 0e41abe35d8f40a2a01953eea1c5d092 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 854 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } }
I20240502 14:25:46.300742 13250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61f17c159094ff59d935ed0ef6381b8" candidate_uuid: "0e41abe35d8f40a2a01953eea1c5d092" candidate_term: 2 candidate_status { last_received { term: 1 index: 942 } } ignore_live_leader: true dest_uuid: "93c82e87d8af478292ffe1a3d3986049"
I20240502 14:25:46.300977 12068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61f17c159094ff59d935ed0ef6381b8" candidate_uuid: "0e41abe35d8f40a2a01953eea1c5d092" candidate_term: 2 candidate_status { last_received { term: 1 index: 942 } } ignore_live_leader: true dest_uuid: "36e66c7b1a03458b89a91c56a57df72a"
I20240502 14:25:46.301111 13250 raft_consensus.cc:3037] T f61f17c159094ff59d935ed0ef6381b8 P 93c82e87d8af478292ffe1a3d3986049 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:25:46.304003 13250 raft_consensus.cc:2454] T f61f17c159094ff59d935ed0ef6381b8 P 93c82e87d8af478292ffe1a3d3986049 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e41abe35d8f40a2a01953eea1c5d092 in term 2.
I20240502 14:25:46.304356 11935 leader_election.cc:288] T f61f17c159094ff59d935ed0ef6381b8 P 0e41abe35d8f40a2a01953eea1c5d092 [CANDIDATE]: Term 2 election: Requested vote from peers 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445), 93c82e87d8af478292ffe1a3d3986049 (127.5.107.68:37273)
I20240502 14:25:46.304405 11871 leader_election.cc:302] T f61f17c159094ff59d935ed0ef6381b8 P 0e41abe35d8f40a2a01953eea1c5d092 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e41abe35d8f40a2a01953eea1c5d092, 93c82e87d8af478292ffe1a3d3986049; no voters:
I20240502 14:25:46.304755 15924 raft_consensus.cc:2781] T f61f17c159094ff59d935ed0ef6381b8 P 0e41abe35d8f40a2a01953eea1c5d092 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:25:46.304869 15924 raft_consensus.cc:686] T f61f17c159094ff59d935ed0ef6381b8 P 0e41abe35d8f40a2a01953eea1c5d092 [term 2 LEADER]: Becoming Leader. State: Replica: 0e41abe35d8f40a2a01953eea1c5d092, State: Running, Role: LEADER
I20240502 14:25:46.305088 15924 consensus_queue.cc:237] T f61f17c159094ff59d935ed0ef6381b8 P 0e41abe35d8f40a2a01953eea1c5d092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 942, Committed index: 942, Last appended: 1.942, Last appended by leader: 942, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 854 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } }
I20240502 14:25:46.305341 12068 raft_consensus.cc:3032] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:25:46.305436 12068 raft_consensus.cc:729] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 36e66c7b1a03458b89a91c56a57df72a, State: Running, Role: LEADER
I20240502 14:25:46.305663 12068 consensus_queue.cc:260] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 942, Committed index: 942, Last appended: 1.942, Last appended by leader: 942, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 854 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } }
I20240502 14:25:46.305951 12068 raft_consensus.cc:3037] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:25:46.308555 12068 raft_consensus.cc:2454] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e41abe35d8f40a2a01953eea1c5d092 in term 2.
W20240502 14:25:46.313807 12026 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32836: Illegal state: replica 36e66c7b1a03458b89a91c56a57df72a is not leader of this config: current role FOLLOWER
I20240502 14:25:46.315291 11785 catalog_manager.cc:5484] T f61f17c159094ff59d935ed0ef6381b8 P 0e41abe35d8f40a2a01953eea1c5d092 reported cstate change: term changed from 1 to 2, leader changed from 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66) to 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65). New cstate: current_term: 2 leader_uuid: "0e41abe35d8f40a2a01953eea1c5d092" committed_config { opid_index: 854 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } } }
I20240502 14:25:46.603706 12068 raft_consensus.cc:1261] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a [term 2 FOLLOWER]: Refusing update from remote peer 0e41abe35d8f40a2a01953eea1c5d092: Log matching property violated. Preceding OpId in replica: term: 1 index: 942. Preceding OpId from leader: term: 2 index: 944. (index mismatch)
I20240502 14:25:46.604180 13248 raft_consensus.cc:1261] T f61f17c159094ff59d935ed0ef6381b8 P 93c82e87d8af478292ffe1a3d3986049 [term 2 FOLLOWER]: Refusing update from remote peer 0e41abe35d8f40a2a01953eea1c5d092: Log matching property violated. Preceding OpId in replica: term: 1 index: 942. Preceding OpId from leader: term: 2 index: 944. (index mismatch)
I20240502 14:25:46.615666 16781 consensus_queue.cc:1035] T f61f17c159094ff59d935ed0ef6381b8 P 0e41abe35d8f40a2a01953eea1c5d092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 943, Last known committed idx: 942, Time since last communication: 0.000s
I20240502 14:25:46.617637 16652 consensus_queue.cc:1035] T f61f17c159094ff59d935ed0ef6381b8 P 0e41abe35d8f40a2a01953eea1c5d092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 943, Last known committed idx: 942, Time since last communication: 0.000s
I20240502 14:25:47.044293 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:47.092370 11916 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:25:47.097257 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:47.116006 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:47.116633 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:47.154175 12045 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:25:47.180480 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:47.727574 11935 consensus_queue.cc:237] T f61f17c159094ff59d935ed0ef6381b8 P 0e41abe35d8f40a2a01953eea1c5d092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 967, Committed index: 967, Last appended: 2.967, Last appended by leader: 942, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 968 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } }
I20240502 14:25:47.733080 13249 raft_consensus.cc:1261] T f61f17c159094ff59d935ed0ef6381b8 P 93c82e87d8af478292ffe1a3d3986049 [term 2 FOLLOWER]: Refusing update from remote peer 0e41abe35d8f40a2a01953eea1c5d092: Log matching property violated. Preceding OpId in replica: term: 2 index: 967. Preceding OpId from leader: term: 2 index: 968. (index mismatch)
I20240502 14:25:47.734467 15924 consensus_queue.cc:1035] T f61f17c159094ff59d935ed0ef6381b8 P 0e41abe35d8f40a2a01953eea1c5d092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 968, Last known committed idx: 966, Time since last communication: 0.000s
I20240502 14:25:47.743438 16652 raft_consensus.cc:2932] T f61f17c159094ff59d935ed0ef6381b8 P 0e41abe35d8f40a2a01953eea1c5d092 [term 2 LEADER]: Committing config change with OpId 2.968: config changed from index 854 to 968, VOTER 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66) evicted. New config: { opid_index: 968 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } } }
I20240502 14:25:47.749058 11784 catalog_manager.cc:5484] T f61f17c159094ff59d935ed0ef6381b8 P 0e41abe35d8f40a2a01953eea1c5d092 reported cstate change: config changed from index 854 to 968, VOTER 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66) evicted. New cstate: current_term: 2 leader_uuid: "0e41abe35d8f40a2a01953eea1c5d092" committed_config { opid_index: 968 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } } }
I20240502 14:25:47.759712 13249 raft_consensus.cc:2932] T f61f17c159094ff59d935ed0ef6381b8 P 93c82e87d8af478292ffe1a3d3986049 [term 2 FOLLOWER]: Committing config change with OpId 2.968: config changed from index 854 to 968, VOTER 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66) evicted. New config: { opid_index: 968 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "93c82e87d8af478292ffe1a3d3986049" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 37273 } } }
I20240502 14:25:47.825722 12045 tablet_service.cc:1514] Processing DeleteTablet for tablet f61f17c159094ff59d935ed0ef6381b8 with delete_type TABLET_DATA_TOMBSTONED (TS 36e66c7b1a03458b89a91c56a57df72a not found in new config with opid_index 968) from {username='slave'} at 127.0.0.1:36164
I20240502 14:25:47.831373 16878 tablet_replica.cc:330] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a: stopping tablet replica
I20240502 14:25:47.831710 16878 raft_consensus.cc:2229] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:25:47.832058 16878 raft_consensus.cc:2258] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:25:47.833099 16878 ts_tablet_manager.cc:1875] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:25:47.869092 16878 ts_tablet_manager.cc:1888] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.967
I20240502 14:25:47.869421 16878 log.cc:1196] T f61f17c159094ff59d935ed0ef6381b8 P 36e66c7b1a03458b89a91c56a57df72a: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/wals/f61f17c159094ff59d935ed0ef6381b8
W20240502 14:25:47.882786 11999 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.69:33193, user_credentials={real_user=slave}} blocked reactor thread for 44818.9us
I20240502 14:25:47.883291 11780 catalog_manager.cc:4830] TS 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445): tablet f61f17c159094ff59d935ed0ef6381b8 (table rebalance_test_table_4 [id=cc8c51baff5f4081aca1cf7908b85057]) successfully deleted
I20240502 14:25:49.097270 11916 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:25:49.198900 12045 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:25:49.234169 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:49.242448 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:49.296190 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:49.320391 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:49.495626 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:50.628924 11916 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:25:50.765375 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:50.777499 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:50.788128 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:50.789944 12045 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:25:50.809244 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:50.819914 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:53.521867 11916 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:25:53.534340 12045 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:25:53.558533 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:53.594946 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:53.749969 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:53.779570 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:53.785019 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:54.988530 12045 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:25:55.015928 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:55.026290 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:55.039316 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:55.064621 11916 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:25:55.102684 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:55.109992 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:55.894404 11916 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:25:55.940428 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:25:55.941452 12045 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:25:55.967073 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:25:55.980558 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:55.985028 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:25:56.009454 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
525d10aecf2c4fbb9c5cdfe9286d0a80 | 127.5.107.126:36087 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.107.84:34899 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0e41abe35d8f40a2a01953eea1c5d092 | 127.5.107.65:36871 | HEALTHY | <none> | 5 | 0
36e66c7b1a03458b89a91c56a57df72a | 127.5.107.66:35445 | HEALTHY | <none> | 5 | 0
375c6d5bdf924866b69b0e6667732692 | 127.5.107.70:33951 | HEALTHY | <none> | 1 | 0
723fb53f34c94c6aa5f7ee3a65711268 | 127.5.107.69:33193 | HEALTHY | <none> | 0 | 0
93c82e87d8af478292ffe1a3d3986049 | 127.5.107.68:37273 | HEALTHY | <none> | 0 | 0
b0d1c78721a746e99d0d3056f1e31371 | 127.5.107.71:33799 | HEALTHY | <none> | 1 | 0
d9748c575342401bb80683c7e2c4e105 | 127.5.107.67:45923 | HEALTHY | <none> | 4 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.5.107.65 | experimental | 127.5.107.65:36871
local_ip_for_outbound_sockets | 127.5.107.66 | experimental | 127.5.107.66:35445
local_ip_for_outbound_sockets | 127.5.107.67 | experimental | 127.5.107.67:45923
local_ip_for_outbound_sockets | 127.5.107.68 | experimental | 127.5.107.68:37273
local_ip_for_outbound_sockets | 127.5.107.69 | experimental | 127.5.107.69:33193
local_ip_for_outbound_sockets | 127.5.107.70 | experimental | 127.5.107.70:33951
local_ip_for_outbound_sockets | 127.5.107.71 | experimental | 127.5.107.71:33799
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.5.107.65:36871
server_dump_info_path | /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.107.66:35445
server_dump_info_path | /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.107.67:45923
server_dump_info_path | /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.5.107.68:37273
server_dump_info_path | /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.107.69:33193
server_dump_info_path | /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.107.70:33951
server_dump_info_path | /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.5.107.71:33799
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.107.84:34899 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 14:25:56.091310 5549 log_verifier.cc:126] Checking tablet 003642773a724534847246de230ea34b
I20240502 14:25:56.599170 5549 log_verifier.cc:177] Verified matching terms for 1474 ops in tablet 003642773a724534847246de230ea34b
I20240502 14:25:56.601068 5549 log_verifier.cc:126] Checking tablet 1427194e32d545f785df3e45137aa42e
I20240502 14:25:56.965811 5549 log_verifier.cc:177] Verified matching terms for 1110 ops in tablet 1427194e32d545f785df3e45137aa42e
I20240502 14:25:56.967105 5549 log_verifier.cc:126] Checking tablet 2f0a86c3cb6c48258f4d2f916b192c9c
I20240502 14:25:57.395480 5549 log_verifier.cc:177] Verified matching terms for 1347 ops in tablet 2f0a86c3cb6c48258f4d2f916b192c9c
I20240502 14:25:57.397145 5549 log_verifier.cc:126] Checking tablet 439f0129f4bd49b7b582be22be83e0b4
I20240502 14:25:57.837147 5549 log_verifier.cc:177] Verified matching terms for 1430 ops in tablet 439f0129f4bd49b7b582be22be83e0b4
I20240502 14:25:57.838764 5549 log_verifier.cc:126] Checking tablet 4987edf6ce4d44d18162ff06e3f74e23
I20240502 14:25:58.169245 5549 log_verifier.cc:177] Verified matching terms for 1031 ops in tablet 4987edf6ce4d44d18162ff06e3f74e23
I20240502 14:25:58.170418 5549 log_verifier.cc:126] Checking tablet 64e9d77a5b9343489db94f4988180213
I20240502 14:25:58.538180 5549 log_verifier.cc:177] Verified matching terms for 1158 ops in tablet 64e9d77a5b9343489db94f4988180213
I20240502 14:25:58.539361 5549 log_verifier.cc:126] Checking tablet 71d8679988d0444c9e0434d7e6f142e8
I20240502 14:25:58.978905 5549 log_verifier.cc:177] Verified matching terms for 1409 ops in tablet 71d8679988d0444c9e0434d7e6f142e8
I20240502 14:25:58.980227 5549 log_verifier.cc:126] Checking tablet 877f199681534650bc1d247cf7608f5f
I20240502 14:25:59.387673 5549 log_verifier.cc:177] Verified matching terms for 1316 ops in tablet 877f199681534650bc1d247cf7608f5f
I20240502 14:25:59.389336 5549 log_verifier.cc:126] Checking tablet 9069116816104c79819b25838836575b
I20240502 14:25:59.784014 5549 log_verifier.cc:177] Verified matching terms for 1225 ops in tablet 9069116816104c79819b25838836575b
I20240502 14:25:59.785173 5549 log_verifier.cc:126] Checking tablet bb1b842f76d74f6e996e48c245deda9b
I20240502 14:26:00.133404 5549 log_verifier.cc:177] Verified matching terms for 1208 ops in tablet bb1b842f76d74f6e996e48c245deda9b
I20240502 14:26:00.134599 5549 log_verifier.cc:126] Checking tablet c3811b0d595f4292b9d96454deba8156
I20240502 14:26:00.477313 5549 log_verifier.cc:177] Verified matching terms for 1022 ops in tablet c3811b0d595f4292b9d96454deba8156
I20240502 14:26:00.478374 5549 log_verifier.cc:126] Checking tablet d55e54737408455fadf1a1b4041edd0a
I20240502 14:26:00.805572 5549 log_verifier.cc:177] Verified matching terms for 1123 ops in tablet d55e54737408455fadf1a1b4041edd0a
I20240502 14:26:00.806735 5549 log_verifier.cc:126] Checking tablet e05a3a4fb76d4c46908945ad0d838509
I20240502 14:26:01.097384 5549 log_verifier.cc:177] Verified matching terms for 1005 ops in tablet e05a3a4fb76d4c46908945ad0d838509
I20240502 14:26:01.098402 5549 log_verifier.cc:126] Checking tablet e7732ff4fa2744db94d0f8cb888de8d8
I20240502 14:26:01.530819 5549 log_verifier.cc:177] Verified matching terms for 1418 ops in tablet e7732ff4fa2744db94d0f8cb888de8d8
I20240502 14:26:01.532228 5549 log_verifier.cc:126] Checking tablet f279550142c648d89b81feee9740f43e
I20240502 14:26:01.547816 5549 log_verifier.cc:177] Verified matching terms for 4 ops in tablet f279550142c648d89b81feee9740f43e
I20240502 14:26:01.547905 5549 log_verifier.cc:126] Checking tablet f61f17c159094ff59d935ed0ef6381b8
I20240502 14:26:01.914395 5549 log_verifier.cc:177] Verified matching terms for 1124 ops in tablet f61f17c159094ff59d935ed0ef6381b8
I20240502 14:26:01.927208 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.72:0
--local_ip_for_outbound_sockets=127.5.107.72
--webserver_interface=127.5.107.72
--webserver_port=0
--tserver_master_addrs=127.5.107.126:36087
--builtin_ntp_servers=127.5.107.84:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 14:26:02.272080 17233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:26:02.272336 17233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:26:02.272423 17233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:26:02.279023 17233 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:26:02.279119 17233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:26:02.279242 17233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.72
I20240502 14:26:02.287088 17233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.5.107.72
--webserver_port=0
--tserver_master_addrs=127.5.107.126:36087
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.72
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:26:02.288255 17233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:26:02.289862 17233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:26:02.301666 17238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:26:02.302212 17239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:26:02.302773 17241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:26:02.303339 17233 server_base.cc:1034] running on GCE node
I20240502 14:26:02.303817 17233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:26:02.304543 17233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:26:02.305728 17233 hybrid_clock.cc:648] HybridClock initialized: now 1714659962305687 us; error 27 us; skew 500 ppm
I20240502 14:26:02.306012 17233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:26:02.307937 17233 webserver.cc:460] Webserver started at http://127.5.107.72:42117/ using document root <none> and password file <none>
I20240502 14:26:02.308588 17233 fs_manager.cc:359] Metadata directory not provided
I20240502 14:26:02.308693 17233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:26:02.308976 17233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:26:02.311456 17233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "0321170b0cf04eac8a1033185d970e12"
format_stamp: "Formatted at 2024-05-02 14:26:02 on dist-test-slave-2cw1"
I20240502 14:26:02.312212 17233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "0321170b0cf04eac8a1033185d970e12"
format_stamp: "Formatted at 2024-05-02 14:26:02 on dist-test-slave-2cw1"
I20240502 14:26:02.316601 17233 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20240502 14:26:02.319716 17247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:02.321225 17233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 14:26:02.321430 17233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "0321170b0cf04eac8a1033185d970e12"
format_stamp: "Formatted at 2024-05-02 14:26:02 on dist-test-slave-2cw1"
I20240502 14:26:02.321609 17233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:26:02.343173 17233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:26:02.343981 17233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:26:02.344225 17233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:26:02.345088 17233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:26:02.346535 17233 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:26:02.346626 17233 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:02.346726 17233 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:26:02.346798 17233 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:02.388092 17233 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.72:35275
I20240502 14:26:02.388165 17363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.72:35275 every 8 connection(s)
I20240502 14:26:02.389350 17233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 14:26:02.394904 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 17233
I20240502 14:26:02.395118 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 14:26:02.401335 17364 heartbeater.cc:347] Connected to a master server at 127.5.107.126:36087
I20240502 14:26:02.401648 17364 heartbeater.cc:464] Registering TS with master...
I20240502 14:26:02.402458 17364 heartbeater.cc:511] Master 127.5.107.126:36087 requested a full tablet report, sending...
I20240502 14:26:02.403932 11787 ts_manager.cc:194] Registered new tserver with Master: 0321170b0cf04eac8a1033185d970e12 (127.5.107.72:35275)
I20240502 14:26:02.404935 11787 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.72:44027
I20240502 14:26:02.862373 11916 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:02.892627 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:02.892655 12045 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:02.892848 17298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:02.913434 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:02.929247 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:02.945132 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:02.985297 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:03.123364 17298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:03.133422 11916 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:03.143604 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:03.151262 12045 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:03.152700 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:03.187036 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:03.220331 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:03.223954 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:03.295080 12200 consensus_queue.cc:237] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1418, Committed index: 1418, Last appended: 2.1418, Last appended by leader: 442, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1419 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } }
I20240502 14:26:03.297401 13653 raft_consensus.cc:1261] T e7732ff4fa2744db94d0f8cb888de8d8 P b0d1c78721a746e99d0d3056f1e31371 [term 2 FOLLOWER]: Refusing update from remote peer d9748c575342401bb80683c7e2c4e105: Log matching property violated. Preceding OpId in replica: term: 2 index: 1418. Preceding OpId from leader: term: 2 index: 1419. (index mismatch)
I20240502 14:26:03.297864 17467 consensus_queue.cc:1035] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1419, Last known committed idx: 1418, Time since last communication: 0.000s
I20240502 14:26:03.300932 17035 raft_consensus.cc:2932] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [term 2 LEADER]: Committing config change with OpId 2.1419: config changed from index 488 to 1419, VOTER 0321170b0cf04eac8a1033185d970e12 (127.5.107.72) added. New config: { opid_index: 1419 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } } }
I20240502 14:26:03.301610 13653 raft_consensus.cc:2932] T e7732ff4fa2744db94d0f8cb888de8d8 P b0d1c78721a746e99d0d3056f1e31371 [term 2 FOLLOWER]: Committing config change with OpId 2.1419: config changed from index 488 to 1419, VOTER 0321170b0cf04eac8a1033185d970e12 (127.5.107.72) added. New config: { opid_index: 1419 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } } }
W20240502 14:26:03.302785 12135 consensus_peers.cc:485] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 -> Peer 0321170b0cf04eac8a1033185d970e12 (127.5.107.72:35275): Couldn't send request to peer 0321170b0cf04eac8a1033185d970e12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e7732ff4fa2744db94d0f8cb888de8d8. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:03.305912 11787 catalog_manager.cc:5484] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 reported cstate change: config changed from index 488 to 1419, VOTER 0321170b0cf04eac8a1033185d970e12 (127.5.107.72) added. New cstate: current_term: 2 leader_uuid: "d9748c575342401bb80683c7e2c4e105" committed_config { opid_index: 1419 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } } }
I20240502 14:26:03.332372 11935 consensus_queue.cc:237] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1316, Committed index: 1316, Last appended: 1.1316, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1317 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } }
I20240502 14:26:03.334532 13381 raft_consensus.cc:1261] T 877f199681534650bc1d247cf7608f5f P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Refusing update from remote peer 0e41abe35d8f40a2a01953eea1c5d092: Log matching property violated. Preceding OpId in replica: term: 1 index: 1316. Preceding OpId from leader: term: 1 index: 1317. (index mismatch)
I20240502 14:26:03.335014 16965 consensus_queue.cc:1035] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1317, Last known committed idx: 1316, Time since last communication: 0.000s
I20240502 14:26:03.337594 17491 raft_consensus.cc:2932] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 LEADER]: Committing config change with OpId 1.1317: config changed from index 519 to 1317, VOTER 0321170b0cf04eac8a1033185d970e12 (127.5.107.72) added. New config: { opid_index: 1317 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } } }
I20240502 14:26:03.338176 13381 raft_consensus.cc:2932] T 877f199681534650bc1d247cf7608f5f P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Committing config change with OpId 1.1317: config changed from index 519 to 1317, VOTER 0321170b0cf04eac8a1033185d970e12 (127.5.107.72) added. New config: { opid_index: 1317 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } } }
W20240502 14:26:03.339596 11871 consensus_peers.cc:485] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 -> Peer 0321170b0cf04eac8a1033185d970e12 (127.5.107.72:35275): Couldn't send request to peer 0321170b0cf04eac8a1033185d970e12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 877f199681534650bc1d247cf7608f5f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:03.343254 11788 catalog_manager.cc:5484] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 reported cstate change: config changed from index 519 to 1317, VOTER 0321170b0cf04eac8a1033185d970e12 (127.5.107.72) added. New cstate: current_term: 1 leader_uuid: "0e41abe35d8f40a2a01953eea1c5d092" committed_config { opid_index: 1317 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } } }
I20240502 14:26:03.376822 12068 consensus_queue.cc:237] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1158, Committed index: 1158, Last appended: 1.1158, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1159 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } }
I20240502 14:26:03.379051 13381 raft_consensus.cc:1261] T 64e9d77a5b9343489db94f4988180213 P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Refusing update from remote peer 36e66c7b1a03458b89a91c56a57df72a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1158. Preceding OpId from leader: term: 1 index: 1159. (index mismatch)
I20240502 14:26:03.379530 17511 consensus_queue.cc:1035] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1159, Last known committed idx: 1158, Time since last communication: 0.000s
I20240502 14:26:03.381990 17224 raft_consensus.cc:2932] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Committing config change with OpId 1.1159: config changed from index 597 to 1159, VOTER 0321170b0cf04eac8a1033185d970e12 (127.5.107.72) added. New config: { opid_index: 1159 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } } }
I20240502 14:26:03.382683 13381 raft_consensus.cc:2932] T 64e9d77a5b9343489db94f4988180213 P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Committing config change with OpId 1.1159: config changed from index 597 to 1159, VOTER 0321170b0cf04eac8a1033185d970e12 (127.5.107.72) added. New config: { opid_index: 1159 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } } }
W20240502 14:26:03.385154 12002 consensus_peers.cc:485] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a -> Peer 0321170b0cf04eac8a1033185d970e12 (127.5.107.72:35275): Couldn't send request to peer 0321170b0cf04eac8a1033185d970e12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 64e9d77a5b9343489db94f4988180213. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:03.386654 11788 catalog_manager.cc:5484] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a reported cstate change: config changed from index 597 to 1159, VOTER 0321170b0cf04eac8a1033185d970e12 (127.5.107.72) added. New cstate: current_term: 1 leader_uuid: "36e66c7b1a03458b89a91c56a57df72a" committed_config { opid_index: 1159 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } } }
I20240502 14:26:03.407480 17364 heartbeater.cc:503] Master 127.5.107.126:36087 was elected leader, sending a full tablet report...
I20240502 14:26:03.415781 12068 consensus_queue.cc:237] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1110, Committed index: 1110, Last appended: 1.1110, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1111 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } }
I20240502 14:26:03.417520 13381 raft_consensus.cc:1261] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Refusing update from remote peer 36e66c7b1a03458b89a91c56a57df72a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1110. Preceding OpId from leader: term: 1 index: 1111. (index mismatch)
W20240502 14:26:03.417594 12002 consensus_peers.cc:485] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a -> Peer 0321170b0cf04eac8a1033185d970e12 (127.5.107.72:35275): Couldn't send request to peer 0321170b0cf04eac8a1033185d970e12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1427194e32d545f785df3e45137aa42e. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:03.418004 17511 consensus_queue.cc:1035] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1111, Last known committed idx: 1110, Time since last communication: 0.000s
I20240502 14:26:03.422156 17511 raft_consensus.cc:2932] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Committing config change with OpId 1.1111: config changed from index 183 to 1111, VOTER 0321170b0cf04eac8a1033185d970e12 (127.5.107.72) added. New config: { opid_index: 1111 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } } }
I20240502 14:26:03.426496 13381 raft_consensus.cc:2932] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Committing config change with OpId 1.1111: config changed from index 183 to 1111, VOTER 0321170b0cf04eac8a1033185d970e12 (127.5.107.72) added. New config: { opid_index: 1111 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } } }
I20240502 14:26:03.431536 11787 catalog_manager.cc:5484] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a reported cstate change: config changed from index 183 to 1111, VOTER 0321170b0cf04eac8a1033185d970e12 (127.5.107.72) added. New cstate: current_term: 1 leader_uuid: "36e66c7b1a03458b89a91c56a57df72a" committed_config { opid_index: 1111 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } } }
I20240502 14:26:03.577710 17298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:03.592499 12045 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:03.592556 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:03.596822 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:03.625646 11916 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:03.630278 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:03.645592 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:03.657634 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:03.887305 11916 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:03.888161 12045 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:03.888656 17298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:03.890269 17600 ts_tablet_manager.cc:902] T e7732ff4fa2744db94d0f8cb888de8d8 P 0321170b0cf04eac8a1033185d970e12: Initiating tablet copy from peer d9748c575342401bb80683c7e2c4e105 (127.5.107.67:45923)
I20240502 14:26:03.912922 17602 ts_tablet_manager.cc:902] T 877f199681534650bc1d247cf7608f5f P 0321170b0cf04eac8a1033185d970e12: Initiating tablet copy from peer 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871)
I20240502 14:26:03.918756 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:03.923616 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:03.957851 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:03.958874 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:03.966084 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:03.994894 17609 ts_tablet_manager.cc:902] T 1427194e32d545f785df3e45137aa42e P 0321170b0cf04eac8a1033185d970e12: Initiating tablet copy from peer 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445)
I20240502 14:26:04.006034 17611 ts_tablet_manager.cc:902] T 64e9d77a5b9343489db94f4988180213 P 0321170b0cf04eac8a1033185d970e12: Initiating tablet copy from peer 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445)
W20240502 14:26:04.025908 13456 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.68:37273, user_credentials={real_user=slave}} blocked reactor thread for 48757.4us
I20240502 14:26:04.039057 17602 tablet_copy_client.cc:308] T 877f199681534650bc1d247cf7608f5f P 0321170b0cf04eac8a1033185d970e12: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.65:36871
I20240502 14:26:04.039150 17609 tablet_copy_client.cc:308] T 1427194e32d545f785df3e45137aa42e P 0321170b0cf04eac8a1033185d970e12: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:35445
I20240502 14:26:04.039057 17600 tablet_copy_client.cc:308] T e7732ff4fa2744db94d0f8cb888de8d8 P 0321170b0cf04eac8a1033185d970e12: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.67:45923
I20240502 14:26:04.067560 12088 tablet_copy_service.cc:140] P 36e66c7b1a03458b89a91c56a57df72a: Received BeginTabletCopySession request for tablet 1427194e32d545f785df3e45137aa42e from peer 0321170b0cf04eac8a1033185d970e12 ({username='slave'} at 127.5.107.72:46537)
I20240502 14:26:04.067755 12088 tablet_copy_service.cc:161] P 36e66c7b1a03458b89a91c56a57df72a: Beginning new tablet copy session on tablet 1427194e32d545f785df3e45137aa42e from peer 0321170b0cf04eac8a1033185d970e12 at {username='slave'} at 127.5.107.72:46537: session id = 0321170b0cf04eac8a1033185d970e12-1427194e32d545f785df3e45137aa42e
I20240502 14:26:04.070823 12088 tablet_copy_source_session.cc:205] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:04.072183 11956 tablet_copy_service.cc:140] P 0e41abe35d8f40a2a01953eea1c5d092: Received BeginTabletCopySession request for tablet 877f199681534650bc1d247cf7608f5f from peer 0321170b0cf04eac8a1033185d970e12 ({username='slave'} at 127.5.107.72:53509)
I20240502 14:26:04.072376 11956 tablet_copy_service.cc:161] P 0e41abe35d8f40a2a01953eea1c5d092: Beginning new tablet copy session on tablet 877f199681534650bc1d247cf7608f5f from peer 0321170b0cf04eac8a1033185d970e12 at {username='slave'} at 127.5.107.72:53509: session id = 0321170b0cf04eac8a1033185d970e12-877f199681534650bc1d247cf7608f5f
I20240502 14:26:04.076479 11956 tablet_copy_source_session.cc:205] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:04.082023 17602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 877f199681534650bc1d247cf7608f5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:04.082732 17609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1427194e32d545f785df3e45137aa42e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:04.094904 17609 tablet_copy_client.cc:783] T 1427194e32d545f785df3e45137aa42e P 0321170b0cf04eac8a1033185d970e12: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:04.095270 17609 tablet_copy_client.cc:647] T 1427194e32d545f785df3e45137aa42e P 0321170b0cf04eac8a1033185d970e12: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:04.095258 12220 tablet_copy_service.cc:140] P d9748c575342401bb80683c7e2c4e105: Received BeginTabletCopySession request for tablet e7732ff4fa2744db94d0f8cb888de8d8 from peer 0321170b0cf04eac8a1033185d970e12 ({username='slave'} at 127.5.107.72:45225)
I20240502 14:26:04.095424 12220 tablet_copy_service.cc:161] P d9748c575342401bb80683c7e2c4e105: Beginning new tablet copy session on tablet e7732ff4fa2744db94d0f8cb888de8d8 from peer 0321170b0cf04eac8a1033185d970e12 at {username='slave'} at 127.5.107.72:45225: session id = 0321170b0cf04eac8a1033185d970e12-e7732ff4fa2744db94d0f8cb888de8d8
I20240502 14:26:04.096258 17602 tablet_copy_client.cc:783] T 877f199681534650bc1d247cf7608f5f P 0321170b0cf04eac8a1033185d970e12: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:04.096509 17602 tablet_copy_client.cc:647] T 877f199681534650bc1d247cf7608f5f P 0321170b0cf04eac8a1033185d970e12: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:04.096962 17611 tablet_copy_client.cc:308] T 64e9d77a5b9343489db94f4988180213 P 0321170b0cf04eac8a1033185d970e12: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:35445
I20240502 14:26:04.097796 12220 tablet_copy_source_session.cc:205] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:04.098440 12088 tablet_copy_service.cc:140] P 36e66c7b1a03458b89a91c56a57df72a: Received BeginTabletCopySession request for tablet 64e9d77a5b9343489db94f4988180213 from peer 0321170b0cf04eac8a1033185d970e12 ({username='slave'} at 127.5.107.72:46537)
I20240502 14:26:04.098595 12088 tablet_copy_service.cc:161] P 36e66c7b1a03458b89a91c56a57df72a: Beginning new tablet copy session on tablet 64e9d77a5b9343489db94f4988180213 from peer 0321170b0cf04eac8a1033185d970e12 at {username='slave'} at 127.5.107.72:46537: session id = 0321170b0cf04eac8a1033185d970e12-64e9d77a5b9343489db94f4988180213
I20240502 14:26:04.099256 17600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7732ff4fa2744db94d0f8cb888de8d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:04.101540 12088 tablet_copy_source_session.cc:205] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:04.104581 17600 tablet_copy_client.cc:783] T e7732ff4fa2744db94d0f8cb888de8d8 P 0321170b0cf04eac8a1033185d970e12: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:04.105782 17600 tablet_copy_client.cc:647] T e7732ff4fa2744db94d0f8cb888de8d8 P 0321170b0cf04eac8a1033185d970e12: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:04.106277 17602 tablet_copy_client.cc:523] T 877f199681534650bc1d247cf7608f5f P 0321170b0cf04eac8a1033185d970e12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:04.106957 17611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64e9d77a5b9343489db94f4988180213. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:04.110879 17602 tablet_bootstrap.cc:492] T 877f199681534650bc1d247cf7608f5f P 0321170b0cf04eac8a1033185d970e12: Bootstrap starting.
I20240502 14:26:04.111977 17611 tablet_copy_client.cc:783] T 64e9d77a5b9343489db94f4988180213 P 0321170b0cf04eac8a1033185d970e12: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:04.112231 17611 tablet_copy_client.cc:647] T 64e9d77a5b9343489db94f4988180213 P 0321170b0cf04eac8a1033185d970e12: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:04.115211 17609 tablet_copy_client.cc:523] T 1427194e32d545f785df3e45137aa42e P 0321170b0cf04eac8a1033185d970e12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:04.116849 17600 tablet_copy_client.cc:523] T e7732ff4fa2744db94d0f8cb888de8d8 P 0321170b0cf04eac8a1033185d970e12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:04.119791 17609 tablet_bootstrap.cc:492] T 1427194e32d545f785df3e45137aa42e P 0321170b0cf04eac8a1033185d970e12: Bootstrap starting.
I20240502 14:26:04.120370 17611 tablet_copy_client.cc:523] T 64e9d77a5b9343489db94f4988180213 P 0321170b0cf04eac8a1033185d970e12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:04.124380 17611 tablet_bootstrap.cc:492] T 64e9d77a5b9343489db94f4988180213 P 0321170b0cf04eac8a1033185d970e12: Bootstrap starting.
I20240502 14:26:04.125589 17600 tablet_bootstrap.cc:492] T e7732ff4fa2744db94d0f8cb888de8d8 P 0321170b0cf04eac8a1033185d970e12: Bootstrap starting.
I20240502 14:26:04.402664 17602 log.cc:824] T 877f199681534650bc1d247cf7608f5f P 0321170b0cf04eac8a1033185d970e12: Log is configured to *not* fsync() on all Append() calls
I20240502 14:26:04.602669 17298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:04.682127 12045 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:04.702347 11916 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:04.716987 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:04.720551 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:04.720839 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:04.744688 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:04.747907 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:05.280622 17298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:05.322530 11916 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:05.346488 12045 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:05.373268 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:05.420320 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:05.485962 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:05.503793 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:05.521502 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:05.946185 17298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:05.976013 11916 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:05.995436 12045 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:06.022293 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:06.032287 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:06.056608 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:06.088747 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:06.112867 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:26:06.203987 12002 consensus_peers.cc:485] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a -> Peer 0321170b0cf04eac8a1033185d970e12 (127.5.107.72:35275): Couldn't send request to peer 0321170b0cf04eac8a1033185d970e12. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240502 14:26:06.312618 11871 consensus_peers.cc:485] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 -> Peer 0321170b0cf04eac8a1033185d970e12 (127.5.107.72:35275): Couldn't send request to peer 0321170b0cf04eac8a1033185d970e12. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240502 14:26:06.441586 12135 consensus_peers.cc:485] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 -> Peer 0321170b0cf04eac8a1033185d970e12 (127.5.107.72:35275): Couldn't send request to peer 0321170b0cf04eac8a1033185d970e12. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240502 14:26:06.467782 17609 tablet_bootstrap.cc:492] T 1427194e32d545f785df3e45137aa42e P 0321170b0cf04eac8a1033185d970e12: Bootstrap replayed 1/1 log segments. Stats: ops{read=1111 overwritten=0 applied=1111 ignored=0} inserts{seen=1740 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:06.468498 17609 tablet_bootstrap.cc:492] T 1427194e32d545f785df3e45137aa42e P 0321170b0cf04eac8a1033185d970e12: Bootstrap complete.
I20240502 14:26:06.469259 17609 ts_tablet_manager.cc:1367] T 1427194e32d545f785df3e45137aa42e P 0321170b0cf04eac8a1033185d970e12: Time spent bootstrapping tablet: real 2.350s user 1.230s sys 0.099s
I20240502 14:26:06.475759 17609 raft_consensus.cc:348] T 1427194e32d545f785df3e45137aa42e P 0321170b0cf04eac8a1033185d970e12 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1111 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } }
I20240502 14:26:06.476312 17609 raft_consensus.cc:729] T 1427194e32d545f785df3e45137aa42e P 0321170b0cf04eac8a1033185d970e12 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0321170b0cf04eac8a1033185d970e12, State: Initialized, Role: FOLLOWER
I20240502 14:26:06.477074 17609 consensus_queue.cc:260] T 1427194e32d545f785df3e45137aa42e P 0321170b0cf04eac8a1033185d970e12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1111, Last appended: 1.1111, Last appended by leader: 1111, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1111 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } }
I20240502 14:26:06.482741 17609 ts_tablet_manager.cc:1398] T 1427194e32d545f785df3e45137aa42e P 0321170b0cf04eac8a1033185d970e12: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20240502 14:26:06.486261 12087 tablet_copy_service.cc:342] P 36e66c7b1a03458b89a91c56a57df72a: Request end of tablet copy session 0321170b0cf04eac8a1033185d970e12-1427194e32d545f785df3e45137aa42e received from {username='slave'} at 127.5.107.72:46537
I20240502 14:26:06.486394 12087 tablet_copy_service.cc:433] P 36e66c7b1a03458b89a91c56a57df72a: ending tablet copy session 0321170b0cf04eac8a1033185d970e12-1427194e32d545f785df3e45137aa42e on tablet 1427194e32d545f785df3e45137aa42e with peer 0321170b0cf04eac8a1033185d970e12
W20240502 14:26:06.530539 12002 consensus_peers.cc:485] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a -> Peer 0321170b0cf04eac8a1033185d970e12 (127.5.107.72:35275): Couldn't send request to peer 0321170b0cf04eac8a1033185d970e12. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240502 14:26:06.560847 17611 tablet_bootstrap.cc:492] T 64e9d77a5b9343489db94f4988180213 P 0321170b0cf04eac8a1033185d970e12: Bootstrap replayed 1/1 log segments. Stats: ops{read=1159 overwritten=0 applied=1159 ignored=0} inserts{seen=1829 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:06.561578 17611 tablet_bootstrap.cc:492] T 64e9d77a5b9343489db94f4988180213 P 0321170b0cf04eac8a1033185d970e12: Bootstrap complete.
I20240502 14:26:06.562119 17611 ts_tablet_manager.cc:1367] T 64e9d77a5b9343489db94f4988180213 P 0321170b0cf04eac8a1033185d970e12: Time spent bootstrapping tablet: real 2.438s user 1.212s sys 0.124s
I20240502 14:26:06.563408 17611 raft_consensus.cc:348] T 64e9d77a5b9343489db94f4988180213 P 0321170b0cf04eac8a1033185d970e12 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1159 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } }
I20240502 14:26:06.563915 17611 raft_consensus.cc:729] T 64e9d77a5b9343489db94f4988180213 P 0321170b0cf04eac8a1033185d970e12 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0321170b0cf04eac8a1033185d970e12, State: Initialized, Role: FOLLOWER
I20240502 14:26:06.564349 17611 consensus_queue.cc:260] T 64e9d77a5b9343489db94f4988180213 P 0321170b0cf04eac8a1033185d970e12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1159, Last appended: 1.1159, Last appended by leader: 1159, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1159 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } }
I20240502 14:26:06.565225 17611 ts_tablet_manager.cc:1398] T 64e9d77a5b9343489db94f4988180213 P 0321170b0cf04eac8a1033185d970e12: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:26:06.565979 12087 tablet_copy_service.cc:342] P 36e66c7b1a03458b89a91c56a57df72a: Request end of tablet copy session 0321170b0cf04eac8a1033185d970e12-64e9d77a5b9343489db94f4988180213 received from {username='slave'} at 127.5.107.72:46537
I20240502 14:26:06.566123 12087 tablet_copy_service.cc:433] P 36e66c7b1a03458b89a91c56a57df72a: ending tablet copy session 0321170b0cf04eac8a1033185d970e12-64e9d77a5b9343489db94f4988180213 on tablet 64e9d77a5b9343489db94f4988180213 with peer 0321170b0cf04eac8a1033185d970e12
I20240502 14:26:06.610857 17298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:06.634325 12045 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:06.653621 11916 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:06.662268 17602 tablet_bootstrap.cc:492] T 877f199681534650bc1d247cf7608f5f P 0321170b0cf04eac8a1033185d970e12: Bootstrap replayed 1/1 log segments. Stats: ops{read=1317 overwritten=0 applied=1317 ignored=0} inserts{seen=2000 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:06.662926 17602 tablet_bootstrap.cc:492] T 877f199681534650bc1d247cf7608f5f P 0321170b0cf04eac8a1033185d970e12: Bootstrap complete.
I20240502 14:26:06.663509 17602 ts_tablet_manager.cc:1367] T 877f199681534650bc1d247cf7608f5f P 0321170b0cf04eac8a1033185d970e12: Time spent bootstrapping tablet: real 2.553s user 1.372s sys 0.145s
I20240502 14:26:06.664724 17602 raft_consensus.cc:348] T 877f199681534650bc1d247cf7608f5f P 0321170b0cf04eac8a1033185d970e12 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1317 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } }
I20240502 14:26:06.665088 17602 raft_consensus.cc:729] T 877f199681534650bc1d247cf7608f5f P 0321170b0cf04eac8a1033185d970e12 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0321170b0cf04eac8a1033185d970e12, State: Initialized, Role: FOLLOWER
I20240502 14:26:06.665560 17602 consensus_queue.cc:260] T 877f199681534650bc1d247cf7608f5f P 0321170b0cf04eac8a1033185d970e12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1317, Last appended: 1.1317, Last appended by leader: 1317, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1317 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } }
I20240502 14:26:06.667200 17602 ts_tablet_manager.cc:1398] T 877f199681534650bc1d247cf7608f5f P 0321170b0cf04eac8a1033185d970e12: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:26:06.668707 11956 tablet_copy_service.cc:342] P 0e41abe35d8f40a2a01953eea1c5d092: Request end of tablet copy session 0321170b0cf04eac8a1033185d970e12-877f199681534650bc1d247cf7608f5f received from {username='slave'} at 127.5.107.72:53509
I20240502 14:26:06.668831 11956 tablet_copy_service.cc:433] P 0e41abe35d8f40a2a01953eea1c5d092: ending tablet copy session 0321170b0cf04eac8a1033185d970e12-877f199681534650bc1d247cf7608f5f on tablet 877f199681534650bc1d247cf7608f5f with peer 0321170b0cf04eac8a1033185d970e12
I20240502 14:26:06.699275 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:06.749557 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:06.759171 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:06.766361 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:06.775527 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:06.803709 17318 raft_consensus.cc:1203] T 1427194e32d545f785df3e45137aa42e P 0321170b0cf04eac8a1033185d970e12 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1110->[1.1111-1.1111] Dedup: 1.1111->[]
I20240502 14:26:06.888844 17318 raft_consensus.cc:1203] T 877f199681534650bc1d247cf7608f5f P 0321170b0cf04eac8a1033185d970e12 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1316->[1.1317-1.1317] Dedup: 1.1317->[]
I20240502 14:26:06.923197 17318 raft_consensus.cc:1203] T 64e9d77a5b9343489db94f4988180213 P 0321170b0cf04eac8a1033185d970e12 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1158->[1.1159-1.1159] Dedup: 1.1159->[]
I20240502 14:26:06.927110 17298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:06.929376 11916 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:06.951774 12045 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:06.965683 17600 tablet_bootstrap.cc:492] T e7732ff4fa2744db94d0f8cb888de8d8 P 0321170b0cf04eac8a1033185d970e12: Bootstrap replayed 1/1 log segments. Stats: ops{read=1419 overwritten=0 applied=1419 ignored=0} inserts{seen=2047 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:06.966149 17600 tablet_bootstrap.cc:492] T e7732ff4fa2744db94d0f8cb888de8d8 P 0321170b0cf04eac8a1033185d970e12: Bootstrap complete.
I20240502 14:26:06.966567 17600 ts_tablet_manager.cc:1367] T e7732ff4fa2744db94d0f8cb888de8d8 P 0321170b0cf04eac8a1033185d970e12: Time spent bootstrapping tablet: real 2.841s user 1.474s sys 0.134s
I20240502 14:26:06.967274 17600 raft_consensus.cc:348] T e7732ff4fa2744db94d0f8cb888de8d8 P 0321170b0cf04eac8a1033185d970e12 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1419 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } }
I20240502 14:26:06.967518 17600 raft_consensus.cc:729] T e7732ff4fa2744db94d0f8cb888de8d8 P 0321170b0cf04eac8a1033185d970e12 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0321170b0cf04eac8a1033185d970e12, State: Initialized, Role: FOLLOWER
I20240502 14:26:06.967725 17600 consensus_queue.cc:260] T e7732ff4fa2744db94d0f8cb888de8d8 P 0321170b0cf04eac8a1033185d970e12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1419, Last appended: 2.1419, Last appended by leader: 1419, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1419 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } }
I20240502 14:26:06.968324 17600 ts_tablet_manager.cc:1398] T e7732ff4fa2744db94d0f8cb888de8d8 P 0321170b0cf04eac8a1033185d970e12: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:26:06.968936 12220 tablet_copy_service.cc:342] P d9748c575342401bb80683c7e2c4e105: Request end of tablet copy session 0321170b0cf04eac8a1033185d970e12-e7732ff4fa2744db94d0f8cb888de8d8 received from {username='slave'} at 127.5.107.72:45225
I20240502 14:26:06.969051 12220 tablet_copy_service.cc:433] P d9748c575342401bb80683c7e2c4e105: ending tablet copy session 0321170b0cf04eac8a1033185d970e12-e7732ff4fa2744db94d0f8cb888de8d8 on tablet e7732ff4fa2744db94d0f8cb888de8d8 with peer 0321170b0cf04eac8a1033185d970e12
I20240502 14:26:06.994524 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:06.997048 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:07.029798 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:07.030398 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:07.053964 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:07.398622 11916 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:07.406134 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:07.431473 13366 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:07.447019 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:07.453143 17318 raft_consensus.cc:1203] T e7732ff4fa2744db94d0f8cb888de8d8 P 0321170b0cf04eac8a1033185d970e12 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.1418->[2.1419-2.1419] Dedup: 2.1419->[]
I20240502 14:26:07.457144 12045 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:07.463119 17298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:07.470655 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:07.471195 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:07.540856 12068 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1427194e32d545f785df3e45137aa42e"
dest_uuid: "36e66c7b1a03458b89a91c56a57df72a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41170
I20240502 14:26:07.541034 12068 raft_consensus.cc:595] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Received request to transfer leadership
I20240502 14:26:07.551270 17877 raft_consensus.cc:979] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a: : Instructing follower 723fb53f34c94c6aa5f7ee3a65711268 to start an election
I20240502 14:26:07.551410 17877 raft_consensus.cc:1067] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Signalling peer 723fb53f34c94c6aa5f7ee3a65711268 to start an election
I20240502 14:26:07.552171 13381 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1427194e32d545f785df3e45137aa42e"
dest_uuid: "723fb53f34c94c6aa5f7ee3a65711268"
from {username='slave'} at 127.5.107.66:57191
I20240502 14:26:07.552467 13381 raft_consensus.cc:482] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:26:07.552603 13381 raft_consensus.cc:3037] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:26:07.555248 13381 raft_consensus.cc:504] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 1111 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } }
I20240502 14:26:07.556828 13381 leader_election.cc:288] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268 [CANDIDATE]: Term 2 election: Requested vote from peers 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445), 0321170b0cf04eac8a1033185d970e12 (127.5.107.72:35275)
I20240502 14:26:07.565922 12068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1427194e32d545f785df3e45137aa42e" candidate_uuid: "723fb53f34c94c6aa5f7ee3a65711268" candidate_term: 2 candidate_status { last_received { term: 1 index: 1111 } } ignore_live_leader: true dest_uuid: "36e66c7b1a03458b89a91c56a57df72a"
I20240502 14:26:07.566164 12068 raft_consensus.cc:3032] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:26:07.566249 12068 raft_consensus.cc:729] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 36e66c7b1a03458b89a91c56a57df72a, State: Running, Role: LEADER
I20240502 14:26:07.566517 12068 consensus_queue.cc:260] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1111, Committed index: 1111, Last appended: 1.1111, Last appended by leader: 1111, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1111 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } }
I20240502 14:26:07.566905 12068 raft_consensus.cc:3037] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:26:07.569586 12068 raft_consensus.cc:2454] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 723fb53f34c94c6aa5f7ee3a65711268 in term 2.
I20240502 14:26:07.570387 13322 leader_election.cc:302] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36e66c7b1a03458b89a91c56a57df72a, 723fb53f34c94c6aa5f7ee3a65711268; no voters:
I20240502 14:26:07.570466 17318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1427194e32d545f785df3e45137aa42e" candidate_uuid: "723fb53f34c94c6aa5f7ee3a65711268" candidate_term: 2 candidate_status { last_received { term: 1 index: 1111 } } ignore_live_leader: true dest_uuid: "0321170b0cf04eac8a1033185d970e12"
I20240502 14:26:07.570822 17318 raft_consensus.cc:3037] T 1427194e32d545f785df3e45137aa42e P 0321170b0cf04eac8a1033185d970e12 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:26:07.571091 17892 raft_consensus.cc:2781] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:26:07.571547 17892 raft_consensus.cc:686] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268 [term 2 LEADER]: Becoming Leader. State: Replica: 723fb53f34c94c6aa5f7ee3a65711268, State: Running, Role: LEADER
I20240502 14:26:07.572034 17892 consensus_queue.cc:237] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1111, Committed index: 1111, Last appended: 1.1111, Last appended by leader: 1111, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1111 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } }
I20240502 14:26:07.573649 17318 raft_consensus.cc:2454] T 1427194e32d545f785df3e45137aa42e P 0321170b0cf04eac8a1033185d970e12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 723fb53f34c94c6aa5f7ee3a65711268 in term 2.
I20240502 14:26:07.574905 11785 catalog_manager.cc:5484] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268 reported cstate change: term changed from 1 to 2, leader changed from 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66) to 723fb53f34c94c6aa5f7ee3a65711268 (127.5.107.69). New cstate: current_term: 2 leader_uuid: "723fb53f34c94c6aa5f7ee3a65711268" committed_config { opid_index: 1111 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } } }
I20240502 14:26:07.647624 17298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:07.655556 11916 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:07.680716 12045 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:07.688637 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:07.694432 13366 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:07.712787 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:07.714708 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:07.731593 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:07.877099 17298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:07.888338 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:07.888967 11916 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:07.889505 12045 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:07.900544 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:07.905201 13366 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:07.907298 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:07.942512 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:07.952412 12068 raft_consensus.cc:1261] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [term 2 FOLLOWER]: Refusing update from remote peer 723fb53f34c94c6aa5f7ee3a65711268: Log matching property violated. Preceding OpId in replica: term: 1 index: 1111. Preceding OpId from leader: term: 2 index: 1112. (index mismatch)
I20240502 14:26:07.953698 17892 consensus_queue.cc:1035] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1112, Last known committed idx: 1111, Time since last communication: 0.000s
I20240502 14:26:07.999346 17318 raft_consensus.cc:1261] T 1427194e32d545f785df3e45137aa42e P 0321170b0cf04eac8a1033185d970e12 [term 2 FOLLOWER]: Refusing update from remote peer 723fb53f34c94c6aa5f7ee3a65711268: Log matching property violated. Preceding OpId in replica: term: 1 index: 1111. Preceding OpId from leader: term: 2 index: 1112. (index mismatch)
I20240502 14:26:08.002990 17893 consensus_queue.cc:1035] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1112, Last known committed idx: 1111, Time since last communication: 0.000s
I20240502 14:26:08.076159 12068 consensus_queue.cc:237] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1159, Committed index: 1159, Last appended: 1.1159, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1160 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } }
I20240502 14:26:08.078043 17318 raft_consensus.cc:1261] T 64e9d77a5b9343489db94f4988180213 P 0321170b0cf04eac8a1033185d970e12 [term 1 FOLLOWER]: Refusing update from remote peer 36e66c7b1a03458b89a91c56a57df72a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1159. Preceding OpId from leader: term: 1 index: 1160. (index mismatch)
I20240502 14:26:08.078521 17877 consensus_queue.cc:1035] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1160, Last known committed idx: 1159, Time since last communication: 0.000s
I20240502 14:26:08.081075 17511 raft_consensus.cc:2932] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a [term 1 LEADER]: Committing config change with OpId 1.1160: config changed from index 1159 to 1160, VOTER 723fb53f34c94c6aa5f7ee3a65711268 (127.5.107.69) evicted. New config: { opid_index: 1160 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } } }
I20240502 14:26:08.081919 17318 raft_consensus.cc:2932] T 64e9d77a5b9343489db94f4988180213 P 0321170b0cf04eac8a1033185d970e12 [term 1 FOLLOWER]: Committing config change with OpId 1.1160: config changed from index 1159 to 1160, VOTER 723fb53f34c94c6aa5f7ee3a65711268 (127.5.107.69) evicted. New config: { opid_index: 1160 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } } }
I20240502 14:26:08.086802 11785 catalog_manager.cc:5484] T 64e9d77a5b9343489db94f4988180213 P 36e66c7b1a03458b89a91c56a57df72a reported cstate change: config changed from index 1159 to 1160, VOTER 723fb53f34c94c6aa5f7ee3a65711268 (127.5.107.69) evicted. New cstate: current_term: 1 leader_uuid: "36e66c7b1a03458b89a91c56a57df72a" committed_config { opid_index: 1160 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } } }
I20240502 14:26:08.097290 13366 tablet_service.cc:1514] Processing DeleteTablet for tablet 64e9d77a5b9343489db94f4988180213 with delete_type TABLET_DATA_TOMBSTONED (TS 723fb53f34c94c6aa5f7ee3a65711268 not found in new config with opid_index 1160) from {username='slave'} at 127.0.0.1:36522
I20240502 14:26:08.098345 17997 tablet_replica.cc:330] T 64e9d77a5b9343489db94f4988180213 P 723fb53f34c94c6aa5f7ee3a65711268: stopping tablet replica
I20240502 14:26:08.098760 17997 raft_consensus.cc:2229] T 64e9d77a5b9343489db94f4988180213 P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:26:08.099079 17997 raft_consensus.cc:2258] T 64e9d77a5b9343489db94f4988180213 P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:26:08.101284 17997 ts_tablet_manager.cc:1875] T 64e9d77a5b9343489db94f4988180213 P 723fb53f34c94c6aa5f7ee3a65711268: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:26:08.106300 17997 ts_tablet_manager.cc:1888] T 64e9d77a5b9343489db94f4988180213 P 723fb53f34c94c6aa5f7ee3a65711268: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1159
I20240502 14:26:08.106472 17997 log.cc:1196] T 64e9d77a5b9343489db94f4988180213 P 723fb53f34c94c6aa5f7ee3a65711268: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal/wals/64e9d77a5b9343489db94f4988180213
I20240502 14:26:08.107295 11777 catalog_manager.cc:4830] TS 723fb53f34c94c6aa5f7ee3a65711268 (127.5.107.69:33193): tablet 64e9d77a5b9343489db94f4988180213 (table rebalance_test_table_1 [id=fe50532b3c3d46b7bdae21e985f105f6]) successfully deleted
I20240502 14:26:08.197120 11916 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:08.204300 12045 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:08.208197 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:08.217355 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:08.224619 17298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:08.231514 13366 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:08.234175 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:08.257459 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:08.331157 12200 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e7732ff4fa2744db94d0f8cb888de8d8"
dest_uuid: "d9748c575342401bb80683c7e2c4e105"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42176
I20240502 14:26:08.331389 12200 raft_consensus.cc:595] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [term 2 LEADER]: Received request to transfer leadership
I20240502 14:26:08.449541 11916 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:08.454581 13366 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:08.479178 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:08.480590 18071 raft_consensus.cc:979] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105: : Instructing follower 0321170b0cf04eac8a1033185d970e12 to start an election
I20240502 14:26:08.480866 17838 raft_consensus.cc:1067] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [term 2 LEADER]: Signalling peer 0321170b0cf04eac8a1033185d970e12 to start an election
I20240502 14:26:08.481775 17318 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e7732ff4fa2744db94d0f8cb888de8d8"
dest_uuid: "0321170b0cf04eac8a1033185d970e12"
from {username='slave'} at 127.5.107.67:33121
I20240502 14:26:08.482124 17318 raft_consensus.cc:482] T e7732ff4fa2744db94d0f8cb888de8d8 P 0321170b0cf04eac8a1033185d970e12 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:26:08.482290 17318 raft_consensus.cc:3037] T e7732ff4fa2744db94d0f8cb888de8d8 P 0321170b0cf04eac8a1033185d970e12 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:26:08.484899 17318 raft_consensus.cc:504] T e7732ff4fa2744db94d0f8cb888de8d8 P 0321170b0cf04eac8a1033185d970e12 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 1419 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } }
I20240502 14:26:08.493063 12045 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:08.497648 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:08.498857 17298 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:08.504169 12178 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:08.506505 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:08.516258 12200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7732ff4fa2744db94d0f8cb888de8d8" candidate_uuid: "0321170b0cf04eac8a1033185d970e12" candidate_term: 3 candidate_status { last_received { term: 2 index: 1419 } } ignore_live_leader: true dest_uuid: "d9748c575342401bb80683c7e2c4e105"
I20240502 14:26:08.516479 12200 raft_consensus.cc:3032] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:26:08.516549 12200 raft_consensus.cc:729] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d9748c575342401bb80683c7e2c4e105, State: Running, Role: LEADER
I20240502 14:26:08.516803 12200 consensus_queue.cc:260] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1419, Committed index: 1419, Last appended: 2.1419, Last appended by leader: 1419, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1419 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } }
I20240502 14:26:08.517146 12200 raft_consensus.cc:3037] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:26:08.518959 17318 leader_election.cc:288] T e7732ff4fa2744db94d0f8cb888de8d8 P 0321170b0cf04eac8a1033185d970e12 [CANDIDATE]: Term 3 election: Requested vote from peers d9748c575342401bb80683c7e2c4e105 (127.5.107.67:45923), b0d1c78721a746e99d0d3056f1e31371 (127.5.107.71:33799)
I20240502 14:26:08.519820 12200 raft_consensus.cc:2454] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0321170b0cf04eac8a1033185d970e12 in term 3.
I20240502 14:26:08.520747 17248 leader_election.cc:302] T e7732ff4fa2744db94d0f8cb888de8d8 P 0321170b0cf04eac8a1033185d970e12 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0321170b0cf04eac8a1033185d970e12, d9748c575342401bb80683c7e2c4e105; no voters:
I20240502 14:26:08.521108 17968 raft_consensus.cc:2781] T e7732ff4fa2744db94d0f8cb888de8d8 P 0321170b0cf04eac8a1033185d970e12 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:26:08.550457 17968 raft_consensus.cc:686] T e7732ff4fa2744db94d0f8cb888de8d8 P 0321170b0cf04eac8a1033185d970e12 [term 3 LEADER]: Becoming Leader. State: Replica: 0321170b0cf04eac8a1033185d970e12, State: Running, Role: LEADER
I20240502 14:26:08.553483 17968 consensus_queue.cc:237] T e7732ff4fa2744db94d0f8cb888de8d8 P 0321170b0cf04eac8a1033185d970e12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1419, Committed index: 1419, Last appended: 2.1419, Last appended by leader: 1419, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1419 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } }
I20240502 14:26:08.568874 13653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7732ff4fa2744db94d0f8cb888de8d8" candidate_uuid: "0321170b0cf04eac8a1033185d970e12" candidate_term: 3 candidate_status { last_received { term: 2 index: 1419 } } ignore_live_leader: true dest_uuid: "b0d1c78721a746e99d0d3056f1e31371"
I20240502 14:26:08.569101 13653 raft_consensus.cc:3037] T e7732ff4fa2744db94d0f8cb888de8d8 P b0d1c78721a746e99d0d3056f1e31371 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:26:08.572024 13653 raft_consensus.cc:2454] T e7732ff4fa2744db94d0f8cb888de8d8 P b0d1c78721a746e99d0d3056f1e31371 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0321170b0cf04eac8a1033185d970e12 in term 3.
I20240502 14:26:08.578789 11785 catalog_manager.cc:5484] T e7732ff4fa2744db94d0f8cb888de8d8 P 0321170b0cf04eac8a1033185d970e12 reported cstate change: term changed from 2 to 3, leader changed from d9748c575342401bb80683c7e2c4e105 (127.5.107.67) to 0321170b0cf04eac8a1033185d970e12 (127.5.107.72). New cstate: current_term: 3 leader_uuid: "0321170b0cf04eac8a1033185d970e12" committed_config { opid_index: 1419 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } } }
I20240502 14:26:08.701033 17298 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:08.703498 11916 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:08.709079 12045 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:08.741845 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:08.746289 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:08.762274 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:08.778692 13366 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:08.796074 12178 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:26:08.846213 11935 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "877f199681534650bc1d247cf7608f5f"
dest_uuid: "0e41abe35d8f40a2a01953eea1c5d092"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52748
I20240502 14:26:08.846493 11935 raft_consensus.cc:595] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:26:08.899746 18138 raft_consensus.cc:979] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092: : Instructing follower 0321170b0cf04eac8a1033185d970e12 to start an election
I20240502 14:26:08.899922 18138 raft_consensus.cc:1067] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 LEADER]: Signalling peer 0321170b0cf04eac8a1033185d970e12 to start an election
I20240502 14:26:08.900811 17318 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "877f199681534650bc1d247cf7608f5f"
dest_uuid: "0321170b0cf04eac8a1033185d970e12"
from {username='slave'} at 127.5.107.65:55487
I20240502 14:26:08.901032 17318 raft_consensus.cc:482] T 877f199681534650bc1d247cf7608f5f P 0321170b0cf04eac8a1033185d970e12 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:26:08.901165 17318 raft_consensus.cc:3037] T 877f199681534650bc1d247cf7608f5f P 0321170b0cf04eac8a1033185d970e12 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:26:08.903183 17318 raft_consensus.cc:504] T 877f199681534650bc1d247cf7608f5f P 0321170b0cf04eac8a1033185d970e12 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 1317 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } }
I20240502 14:26:08.903837 17318 leader_election.cc:288] T 877f199681534650bc1d247cf7608f5f P 0321170b0cf04eac8a1033185d970e12 [CANDIDATE]: Term 2 election: Requested vote from peers 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871), 723fb53f34c94c6aa5f7ee3a65711268 (127.5.107.69:33193)
I20240502 14:26:08.904227 11935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "877f199681534650bc1d247cf7608f5f" candidate_uuid: "0321170b0cf04eac8a1033185d970e12" candidate_term: 2 candidate_status { last_received { term: 1 index: 1317 } } ignore_live_leader: true dest_uuid: "0e41abe35d8f40a2a01953eea1c5d092"
I20240502 14:26:08.904421 11935 raft_consensus.cc:3032] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:26:08.904518 11935 raft_consensus.cc:729] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0e41abe35d8f40a2a01953eea1c5d092, State: Running, Role: LEADER
I20240502 14:26:08.904788 11935 consensus_queue.cc:260] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1317, Committed index: 1317, Last appended: 1.1317, Last appended by leader: 1317, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1317 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } }
I20240502 14:26:08.905169 11935 raft_consensus.cc:3037] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:26:08.907656 11935 raft_consensus.cc:2454] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0321170b0cf04eac8a1033185d970e12 in term 2.
I20240502 14:26:08.908219 17249 leader_election.cc:302] T 877f199681534650bc1d247cf7608f5f P 0321170b0cf04eac8a1033185d970e12 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0321170b0cf04eac8a1033185d970e12, 0e41abe35d8f40a2a01953eea1c5d092; no voters:
I20240502 14:26:08.908520 17968 raft_consensus.cc:2781] T 877f199681534650bc1d247cf7608f5f P 0321170b0cf04eac8a1033185d970e12 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:26:08.908666 17968 raft_consensus.cc:686] T 877f199681534650bc1d247cf7608f5f P 0321170b0cf04eac8a1033185d970e12 [term 2 LEADER]: Becoming Leader. State: Replica: 0321170b0cf04eac8a1033185d970e12, State: Running, Role: LEADER
I20240502 14:26:08.908888 17968 consensus_queue.cc:237] T 877f199681534650bc1d247cf7608f5f P 0321170b0cf04eac8a1033185d970e12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1317, Committed index: 1317, Last appended: 1.1317, Last appended by leader: 1317, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1317 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } }
I20240502 14:26:08.910776 13381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "877f199681534650bc1d247cf7608f5f" candidate_uuid: "0321170b0cf04eac8a1033185d970e12" candidate_term: 2 candidate_status { last_received { term: 1 index: 1317 } } ignore_live_leader: true dest_uuid: "723fb53f34c94c6aa5f7ee3a65711268"
I20240502 14:26:08.911002 13381 raft_consensus.cc:3037] T 877f199681534650bc1d247cf7608f5f P 723fb53f34c94c6aa5f7ee3a65711268 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:26:08.913772 13381 raft_consensus.cc:2454] T 877f199681534650bc1d247cf7608f5f P 723fb53f34c94c6aa5f7ee3a65711268 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0321170b0cf04eac8a1033185d970e12 in term 2.
I20240502 14:26:08.928736 11787 catalog_manager.cc:5484] T 877f199681534650bc1d247cf7608f5f P 0321170b0cf04eac8a1033185d970e12 reported cstate change: term changed from 1 to 2, leader changed from 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65) to 0321170b0cf04eac8a1033185d970e12 (127.5.107.72). New cstate: current_term: 2 leader_uuid: "0321170b0cf04eac8a1033185d970e12" committed_config { opid_index: 1317 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } } }
I20240502 14:26:08.958652 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:08.959589 17298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:26:08.963689 11916 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:08.975343 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:08.984612 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:08.989711 12178 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:26:08.997630 12045 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:08.998615 13366 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:09.015348 12200 raft_consensus.cc:1261] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [term 3 FOLLOWER]: Refusing update from remote peer 0321170b0cf04eac8a1033185d970e12: Log matching property violated. Preceding OpId in replica: term: 2 index: 1419. Preceding OpId from leader: term: 3 index: 1420. (index mismatch)
I20240502 14:26:09.016067 17968 consensus_queue.cc:1035] T e7732ff4fa2744db94d0f8cb888de8d8 P 0321170b0cf04eac8a1033185d970e12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1420, Last known committed idx: 1419, Time since last communication: 0.000s
I20240502 14:26:09.022156 13653 raft_consensus.cc:1261] T e7732ff4fa2744db94d0f8cb888de8d8 P b0d1c78721a746e99d0d3056f1e31371 [term 3 FOLLOWER]: Refusing update from remote peer 0321170b0cf04eac8a1033185d970e12: Log matching property violated. Preceding OpId in replica: term: 2 index: 1419. Preceding OpId from leader: term: 3 index: 1420. (index mismatch)
I20240502 14:26:09.022970 17968 consensus_queue.cc:1035] T e7732ff4fa2744db94d0f8cb888de8d8 P 0321170b0cf04eac8a1033185d970e12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1420, Last known committed idx: 1419, Time since last communication: 0.000s
I20240502 14:26:09.324395 11935 raft_consensus.cc:1261] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [term 2 FOLLOWER]: Refusing update from remote peer 0321170b0cf04eac8a1033185d970e12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1317. Preceding OpId from leader: term: 2 index: 1318. (index mismatch)
I20240502 14:26:09.325088 17968 consensus_queue.cc:1035] T 877f199681534650bc1d247cf7608f5f P 0321170b0cf04eac8a1033185d970e12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1318, Last known committed idx: 1317, Time since last communication: 0.000s
I20240502 14:26:09.328886 13381 raft_consensus.cc:1261] T 877f199681534650bc1d247cf7608f5f P 723fb53f34c94c6aa5f7ee3a65711268 [term 2 FOLLOWER]: Refusing update from remote peer 0321170b0cf04eac8a1033185d970e12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1317. Preceding OpId from leader: term: 2 index: 1318. (index mismatch)
I20240502 14:26:09.329708 18073 consensus_queue.cc:1035] T 877f199681534650bc1d247cf7608f5f P 0321170b0cf04eac8a1033185d970e12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1318, Last known committed idx: 1317, Time since last communication: 0.000s
I20240502 14:26:09.465688 17298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:26:09.473666 12045 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:09.500888 11916 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:09.537688 13366 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:09.564532 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:09.575942 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:09.582132 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:09.632318 12178 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:26:09.694422 13381 consensus_queue.cc:237] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1112, Committed index: 1112, Last appended: 2.1112, Last appended by leader: 1111, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1113 OBSOLETE_local: false peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } }
I20240502 14:26:09.696198 17318 raft_consensus.cc:1261] T 1427194e32d545f785df3e45137aa42e P 0321170b0cf04eac8a1033185d970e12 [term 2 FOLLOWER]: Refusing update from remote peer 723fb53f34c94c6aa5f7ee3a65711268: Log matching property violated. Preceding OpId in replica: term: 2 index: 1112. Preceding OpId from leader: term: 2 index: 1113. (index mismatch)
I20240502 14:26:09.696647 18158 consensus_queue.cc:1035] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1113, Last known committed idx: 1112, Time since last communication: 0.000s
I20240502 14:26:09.699199 18227 raft_consensus.cc:2932] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268 [term 2 LEADER]: Committing config change with OpId 2.1113: config changed from index 1111 to 1113, VOTER 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66) evicted. New config: { opid_index: 1113 OBSOLETE_local: false peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } } }
I20240502 14:26:09.699661 17318 raft_consensus.cc:2932] T 1427194e32d545f785df3e45137aa42e P 0321170b0cf04eac8a1033185d970e12 [term 2 FOLLOWER]: Committing config change with OpId 2.1113: config changed from index 1111 to 1113, VOTER 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66) evicted. New config: { opid_index: 1113 OBSOLETE_local: false peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } } }
I20240502 14:26:09.704754 11785 catalog_manager.cc:5484] T 1427194e32d545f785df3e45137aa42e P 723fb53f34c94c6aa5f7ee3a65711268 reported cstate change: config changed from index 1111 to 1113, VOTER 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66) evicted. New cstate: current_term: 2 leader_uuid: "723fb53f34c94c6aa5f7ee3a65711268" committed_config { opid_index: 1113 OBSOLETE_local: false peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } } }
I20240502 14:26:09.709329 12045 tablet_service.cc:1514] Processing DeleteTablet for tablet 1427194e32d545f785df3e45137aa42e with delete_type TABLET_DATA_TOMBSTONED (TS 36e66c7b1a03458b89a91c56a57df72a not found in new config with opid_index 1113) from {username='slave'} at 127.0.0.1:36164
I20240502 14:26:09.709864 18236 tablet_replica.cc:330] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a: stopping tablet replica
I20240502 14:26:09.710045 18236 raft_consensus.cc:2229] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:26:09.710311 18236 raft_consensus.cc:2258] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:26:09.712280 18236 ts_tablet_manager.cc:1875] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:26:09.717731 18236 ts_tablet_manager.cc:1888] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1112
I20240502 14:26:09.717867 18236 log.cc:1196] T 1427194e32d545f785df3e45137aa42e P 36e66c7b1a03458b89a91c56a57df72a: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/wals/1427194e32d545f785df3e45137aa42e
I20240502 14:26:09.718533 11780 catalog_manager.cc:4830] TS 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445): tablet 1427194e32d545f785df3e45137aa42e (table rebalance_test_table_4 [id=cc8c51baff5f4081aca1cf7908b85057]) successfully deleted
I20240502 14:26:09.801316 17298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:26:09.812769 12045 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:09.817407 11916 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:09.823925 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:09.844715 13366 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:09.877396 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:09.887907 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:09.890774 12178 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:26:09.990146 17318 consensus_queue.cc:237] T e7732ff4fa2744db94d0f8cb888de8d8 P 0321170b0cf04eac8a1033185d970e12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1420, Committed index: 1420, Last appended: 3.1420, Last appended by leader: 1419, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1421 OBSOLETE_local: false peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } }
I20240502 14:26:09.991564 13656 raft_consensus.cc:1261] T e7732ff4fa2744db94d0f8cb888de8d8 P b0d1c78721a746e99d0d3056f1e31371 [term 3 FOLLOWER]: Refusing update from remote peer 0321170b0cf04eac8a1033185d970e12: Log matching property violated. Preceding OpId in replica: term: 3 index: 1420. Preceding OpId from leader: term: 3 index: 1421. (index mismatch)
I20240502 14:26:09.992081 18161 consensus_queue.cc:1035] T e7732ff4fa2744db94d0f8cb888de8d8 P 0321170b0cf04eac8a1033185d970e12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1421, Last known committed idx: 1420, Time since last communication: 0.000s
I20240502 14:26:09.994171 18291 raft_consensus.cc:2932] T e7732ff4fa2744db94d0f8cb888de8d8 P 0321170b0cf04eac8a1033185d970e12 [term 3 LEADER]: Committing config change with OpId 3.1421: config changed from index 1419 to 1421, VOTER d9748c575342401bb80683c7e2c4e105 (127.5.107.67) evicted. New config: { opid_index: 1421 OBSOLETE_local: false peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } } }
I20240502 14:26:09.994693 13656 raft_consensus.cc:2932] T e7732ff4fa2744db94d0f8cb888de8d8 P b0d1c78721a746e99d0d3056f1e31371 [term 3 FOLLOWER]: Committing config change with OpId 3.1421: config changed from index 1419 to 1421, VOTER d9748c575342401bb80683c7e2c4e105 (127.5.107.67) evicted. New config: { opid_index: 1421 OBSOLETE_local: false peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } } }
I20240502 14:26:09.999019 11787 catalog_manager.cc:5484] T e7732ff4fa2744db94d0f8cb888de8d8 P b0d1c78721a746e99d0d3056f1e31371 reported cstate change: config changed from index 1419 to 1421, VOTER d9748c575342401bb80683c7e2c4e105 (127.5.107.67) evicted. New cstate: current_term: 3 leader_uuid: "0321170b0cf04eac8a1033185d970e12" committed_config { opid_index: 1421 OBSOLETE_local: false peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } } }
I20240502 14:26:10.003201 12178 tablet_service.cc:1514] Processing DeleteTablet for tablet e7732ff4fa2744db94d0f8cb888de8d8 with delete_type TABLET_DATA_TOMBSTONED (TS d9748c575342401bb80683c7e2c4e105 not found in new config with opid_index 1421) from {username='slave'} at 127.0.0.1:45972
I20240502 14:26:10.005695 18292 tablet_replica.cc:330] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105: stopping tablet replica
I20240502 14:26:10.005964 18292 raft_consensus.cc:2229] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 14:26:10.006181 18292 raft_consensus.cc:2258] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:26:10.007783 18292 ts_tablet_manager.cc:1875] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:26:10.013022 18292 ts_tablet_manager.cc:1888] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.1420
I20240502 14:26:10.013127 18292 log.cc:1196] T e7732ff4fa2744db94d0f8cb888de8d8 P d9748c575342401bb80683c7e2c4e105: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal/wals/e7732ff4fa2744db94d0f8cb888de8d8
I20240502 14:26:10.013918 11777 catalog_manager.cc:4830] TS d9748c575342401bb80683c7e2c4e105 (127.5.107.67:45923): tablet e7732ff4fa2744db94d0f8cb888de8d8 (table rebalance_test_table_2 [id=ec56bc8cfbc04f26b92384c442eac5b8]) successfully deleted
I20240502 14:26:10.107684 11916 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:10.109309 17298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:26:10.112159 12045 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:10.112859 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:10.115833 13366 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:10.117580 12178 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:26:10.142721 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:10.162532 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:10.233165 17318 consensus_queue.cc:237] T 877f199681534650bc1d247cf7608f5f P 0321170b0cf04eac8a1033185d970e12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1318, Committed index: 1318, Last appended: 2.1318, Last appended by leader: 1317, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1319 OBSOLETE_local: false peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } }
I20240502 14:26:10.234617 13381 raft_consensus.cc:1261] T 877f199681534650bc1d247cf7608f5f P 723fb53f34c94c6aa5f7ee3a65711268 [term 2 FOLLOWER]: Refusing update from remote peer 0321170b0cf04eac8a1033185d970e12: Log matching property violated. Preceding OpId in replica: term: 2 index: 1318. Preceding OpId from leader: term: 2 index: 1319. (index mismatch)
I20240502 14:26:10.235078 18161 consensus_queue.cc:1035] T 877f199681534650bc1d247cf7608f5f P 0321170b0cf04eac8a1033185d970e12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1319, Last known committed idx: 1318, Time since last communication: 0.000s
I20240502 14:26:10.236742 18291 raft_consensus.cc:2932] T 877f199681534650bc1d247cf7608f5f P 0321170b0cf04eac8a1033185d970e12 [term 2 LEADER]: Committing config change with OpId 2.1319: config changed from index 1317 to 1319, VOTER 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65) evicted. New config: { opid_index: 1319 OBSOLETE_local: false peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } } }
I20240502 14:26:10.237215 13381 raft_consensus.cc:2932] T 877f199681534650bc1d247cf7608f5f P 723fb53f34c94c6aa5f7ee3a65711268 [term 2 FOLLOWER]: Committing config change with OpId 2.1319: config changed from index 1317 to 1319, VOTER 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65) evicted. New config: { opid_index: 1319 OBSOLETE_local: false peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } } }
I20240502 14:26:10.240855 11787 catalog_manager.cc:5484] T 877f199681534650bc1d247cf7608f5f P 0321170b0cf04eac8a1033185d970e12 reported cstate change: config changed from index 1317 to 1319, VOTER 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65) evicted. New cstate: current_term: 2 leader_uuid: "0321170b0cf04eac8a1033185d970e12" committed_config { opid_index: 1319 OBSOLETE_local: false peers { permanent_uuid: "723fb53f34c94c6aa5f7ee3a65711268" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 33193 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } } }
I20240502 14:26:10.244675 11916 tablet_service.cc:1514] Processing DeleteTablet for tablet 877f199681534650bc1d247cf7608f5f with delete_type TABLET_DATA_TOMBSTONED (TS 0e41abe35d8f40a2a01953eea1c5d092 not found in new config with opid_index 1319) from {username='slave'} at 127.0.0.1:42830
I20240502 14:26:10.245242 18353 tablet_replica.cc:330] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092: stopping tablet replica
I20240502 14:26:10.245431 18353 raft_consensus.cc:2229] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:26:10.245571 18353 raft_consensus.cc:2258] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:26:10.247290 18353 ts_tablet_manager.cc:1875] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:26:10.252763 18353 ts_tablet_manager.cc:1888] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1318
I20240502 14:26:10.252894 18353 log.cc:1196] T 877f199681534650bc1d247cf7608f5f P 0e41abe35d8f40a2a01953eea1c5d092: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal/wals/877f199681534650bc1d247cf7608f5f
I20240502 14:26:10.253676 11778 catalog_manager.cc:4830] TS 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871): tablet 877f199681534650bc1d247cf7608f5f (table rebalance_test_table_3 [id=423ad8739bd34e329c45d60dce8f9687]) successfully deleted
I20240502 14:26:10.357921 11916 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:10.366422 12045 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:10.384876 13366 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:10.404053 17298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:26:10.416725 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:10.417042 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:10.425940 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:10.437767 12178 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:26:10.598964 17298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:26:10.609073 11916 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:10.626859 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:10.628324 12045 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:10.634049 13366 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:10.638911 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:10.639204 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:10.685878 12178 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:26:11.025651 17298 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:26:11.029150 11916 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:11.045277 12045 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:26:11.064910 13366 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:11.099922 13637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:11.103492 12178 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:26:11.105769 13501 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:26:11.111693 13231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
525d10aecf2c4fbb9c5cdfe9286d0a80 | 127.5.107.126:36087 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.107.84:34899 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0321170b0cf04eac8a1033185d970e12 | 127.5.107.72:35275 | HEALTHY | <none> | 2 | 0
0e41abe35d8f40a2a01953eea1c5d092 | 127.5.107.65:36871 | HEALTHY | <none> | 4 | 0
36e66c7b1a03458b89a91c56a57df72a | 127.5.107.66:35445 | HEALTHY | <none> | 4 | 0
375c6d5bdf924866b69b0e6667732692 | 127.5.107.70:33951 | HEALTHY | <none> | 1 | 0
723fb53f34c94c6aa5f7ee3a65711268 | 127.5.107.69:33193 | HEALTHY | <none> | 1 | 0
93c82e87d8af478292ffe1a3d3986049 | 127.5.107.68:37273 | HEALTHY | <none> | 0 | 0
b0d1c78721a746e99d0d3056f1e31371 | 127.5.107.71:33799 | HEALTHY | <none> | 1 | 0
d9748c575342401bb80683c7e2c4e105 | 127.5.107.67:45923 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.5.107.65 | experimental | 127.5.107.65:36871
local_ip_for_outbound_sockets | 127.5.107.66 | experimental | 127.5.107.66:35445
local_ip_for_outbound_sockets | 127.5.107.67 | experimental | 127.5.107.67:45923
local_ip_for_outbound_sockets | 127.5.107.68 | experimental | 127.5.107.68:37273
local_ip_for_outbound_sockets | 127.5.107.69 | experimental | 127.5.107.69:33193
local_ip_for_outbound_sockets | 127.5.107.70 | experimental | 127.5.107.70:33951
local_ip_for_outbound_sockets | 127.5.107.71 | experimental | 127.5.107.71:33799
local_ip_for_outbound_sockets | 127.5.107.72 | experimental | 127.5.107.72:35275
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.5.107.65:36871
server_dump_info_path | /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.107.66:35445
server_dump_info_path | /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.107.67:45923
server_dump_info_path | /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.5.107.68:37273
server_dump_info_path | /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.107.69:33193
server_dump_info_path | /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.107.70:33951
server_dump_info_path | /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.5.107.71:33799
server_dump_info_path | /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.5.107.72:35275
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.107.84:34899 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 4
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 14:26:11.186769 5549 log_verifier.cc:126] Checking tablet 003642773a724534847246de230ea34b
I20240502 14:26:11.621785 5549 log_verifier.cc:177] Verified matching terms for 1474 ops in tablet 003642773a724534847246de230ea34b
I20240502 14:26:11.623584 5549 log_verifier.cc:126] Checking tablet 1427194e32d545f785df3e45137aa42e
I20240502 14:26:11.984560 5549 log_verifier.cc:177] Verified matching terms for 1113 ops in tablet 1427194e32d545f785df3e45137aa42e
I20240502 14:26:11.985893 5549 log_verifier.cc:126] Checking tablet 2f0a86c3cb6c48258f4d2f916b192c9c
I20240502 14:26:12.394903 5549 log_verifier.cc:177] Verified matching terms for 1347 ops in tablet 2f0a86c3cb6c48258f4d2f916b192c9c
I20240502 14:26:12.396231 5549 log_verifier.cc:126] Checking tablet 439f0129f4bd49b7b582be22be83e0b4
I20240502 14:26:12.808483 5549 log_verifier.cc:177] Verified matching terms for 1430 ops in tablet 439f0129f4bd49b7b582be22be83e0b4
I20240502 14:26:12.809962 5549 log_verifier.cc:126] Checking tablet 4987edf6ce4d44d18162ff06e3f74e23
I20240502 14:26:13.090029 5549 log_verifier.cc:177] Verified matching terms for 1031 ops in tablet 4987edf6ce4d44d18162ff06e3f74e23
I20240502 14:26:13.091063 5549 log_verifier.cc:126] Checking tablet 64e9d77a5b9343489db94f4988180213
I20240502 14:26:13.439329 5549 log_verifier.cc:177] Verified matching terms for 1160 ops in tablet 64e9d77a5b9343489db94f4988180213
I20240502 14:26:13.440474 5549 log_verifier.cc:126] Checking tablet 71d8679988d0444c9e0434d7e6f142e8
I20240502 14:26:13.814837 5549 log_verifier.cc:177] Verified matching terms for 1409 ops in tablet 71d8679988d0444c9e0434d7e6f142e8
I20240502 14:26:13.816478 5549 log_verifier.cc:126] Checking tablet 877f199681534650bc1d247cf7608f5f
I20240502 14:26:14.175151 5549 log_verifier.cc:177] Verified matching terms for 1319 ops in tablet 877f199681534650bc1d247cf7608f5f
I20240502 14:26:14.176446 5549 log_verifier.cc:126] Checking tablet 9069116816104c79819b25838836575b
I20240502 14:26:14.546433 5549 log_verifier.cc:177] Verified matching terms for 1225 ops in tablet 9069116816104c79819b25838836575b
I20240502 14:26:14.547761 5549 log_verifier.cc:126] Checking tablet bb1b842f76d74f6e996e48c245deda9b
I20240502 14:26:14.859608 5549 log_verifier.cc:177] Verified matching terms for 1208 ops in tablet bb1b842f76d74f6e996e48c245deda9b
I20240502 14:26:14.860767 5549 log_verifier.cc:126] Checking tablet c3811b0d595f4292b9d96454deba8156
I20240502 14:26:15.169823 5549 log_verifier.cc:177] Verified matching terms for 1022 ops in tablet c3811b0d595f4292b9d96454deba8156
I20240502 14:26:15.170899 5549 log_verifier.cc:126] Checking tablet d55e54737408455fadf1a1b4041edd0a
I20240502 14:26:15.490147 5549 log_verifier.cc:177] Verified matching terms for 1123 ops in tablet d55e54737408455fadf1a1b4041edd0a
I20240502 14:26:15.491251 5549 log_verifier.cc:126] Checking tablet e05a3a4fb76d4c46908945ad0d838509
I20240502 14:26:15.755826 5549 log_verifier.cc:177] Verified matching terms for 1005 ops in tablet e05a3a4fb76d4c46908945ad0d838509
I20240502 14:26:15.756899 5549 log_verifier.cc:126] Checking tablet e7732ff4fa2744db94d0f8cb888de8d8
I20240502 14:26:16.178795 5549 log_verifier.cc:177] Verified matching terms for 1421 ops in tablet e7732ff4fa2744db94d0f8cb888de8d8
I20240502 14:26:16.180326 5549 log_verifier.cc:126] Checking tablet f279550142c648d89b81feee9740f43e
I20240502 14:26:16.191150 5549 log_verifier.cc:177] Verified matching terms for 4 ops in tablet f279550142c648d89b81feee9740f43e
I20240502 14:26:16.191218 5549 log_verifier.cc:126] Checking tablet f61f17c159094ff59d935ed0ef6381b8
I20240502 14:26:16.522342 5549 log_verifier.cc:177] Verified matching terms for 1124 ops in tablet f61f17c159094ff59d935ed0ef6381b8
I20240502 14:26:16.526906 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 11853
I20240502 14:26:16.545658 11976 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:26:16.865871 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 11853
W20240502 14:26:16.912672 13187 connection.cc:535] server connection from 127.5.107.65:57277 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:26:16.913269 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 11984
I20240502 14:26:16.918306 12108 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:26:17.240015 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 11984
W20240502 14:26:17.289276 17249 connection.cc:535] server connection from 127.5.107.66:39459 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:26:17.289848 11778 connection.cc:535] server connection from 127.5.107.66:37535 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:26:17.290226 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 12116
I20240502 14:26:17.295401 12240 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:26:17.612071 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 12116
W20240502 14:26:17.654249 13185 connection.cc:535] server connection from 127.5.107.67:39303 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:26:17.654810 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 13169
I20240502 14:26:17.660296 13292 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:26:17.911375 18496 raft_consensus.cc:482] T f61f17c159094ff59d935ed0ef6381b8 P 93c82e87d8af478292ffe1a3d3986049 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0e41abe35d8f40a2a01953eea1c5d092)
I20240502 14:26:17.911635 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 13169
W20240502 14:26:17.956806 11780 connection.cc:535] server connection from 127.5.107.68:39919 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:26:17.957023 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 13301
I20240502 14:26:17.963258 13428 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:26:17.989267 18499 raft_consensus.cc:482] T bb1b842f76d74f6e996e48c245deda9b P 375c6d5bdf924866b69b0e6667732692 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0e41abe35d8f40a2a01953eea1c5d092)
I20240502 14:26:17.989481 18499 raft_consensus.cc:504] T bb1b842f76d74f6e996e48c245deda9b P 375c6d5bdf924866b69b0e6667732692 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 497 OBSOLETE_local: false peers { permanent_uuid: "0e41abe35d8f40a2a01953eea1c5d092" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36871 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:26:17.990084 18499 leader_election.cc:288] T bb1b842f76d74f6e996e48c245deda9b P 375c6d5bdf924866b69b0e6667732692 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871)
W20240502 14:26:17.990921 13454 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.65:36871: connect: Connection refused (error 111)
W20240502 14:26:17.992182 13454 leader_election.cc:334] T bb1b842f76d74f6e996e48c245deda9b P 375c6d5bdf924866b69b0e6667732692 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0e41abe35d8f40a2a01953eea1c5d092 (127.5.107.65:36871): Network error: Client connection negotiation failed: client connection to 127.5.107.65:36871: connect: Connection refused (error 111)
I20240502 14:26:17.992341 13454 leader_election.cc:302] T bb1b842f76d74f6e996e48c245deda9b P 375c6d5bdf924866b69b0e6667732692 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 375c6d5bdf924866b69b0e6667732692; no voters: 0e41abe35d8f40a2a01953eea1c5d092
I20240502 14:26:17.992650 18499 raft_consensus.cc:2726] T bb1b842f76d74f6e996e48c245deda9b P 375c6d5bdf924866b69b0e6667732692 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:26:17.994328 13456 consensus_peers.cc:485] T f279550142c648d89b81feee9740f43e P 375c6d5bdf924866b69b0e6667732692 -> Peer 93c82e87d8af478292ffe1a3d3986049 (127.5.107.68:37273): Couldn't send request to peer 93c82e87d8af478292ffe1a3d3986049. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.68:37273: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:26:18.166708 13589 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.67:45923: connect: Connection refused (error 111)
W20240502 14:26:18.168694 13589 consensus_peers.cc:485] T d55e54737408455fadf1a1b4041edd0a P b0d1c78721a746e99d0d3056f1e31371 -> Peer d9748c575342401bb80683c7e2c4e105 (127.5.107.67:45923): Couldn't send request to peer d9748c575342401bb80683c7e2c4e105. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.67:45923: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:18.228704 18499 raft_consensus.cc:482] T 003642773a724534847246de230ea34b P 375c6d5bdf924866b69b0e6667732692 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 36e66c7b1a03458b89a91c56a57df72a)
I20240502 14:26:18.228886 18499 raft_consensus.cc:504] T 003642773a724534847246de230ea34b P 375c6d5bdf924866b69b0e6667732692 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 1056 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "375c6d5bdf924866b69b0e6667732692" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 33951 } }
I20240502 14:26:18.229491 18499 leader_election.cc:288] T 003642773a724534847246de230ea34b P 375c6d5bdf924866b69b0e6667732692 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445)
W20240502 14:26:18.230700 13456 leader_election.cc:334] T 003642773a724534847246de230ea34b P 375c6d5bdf924866b69b0e6667732692 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445): Network error: Client connection negotiation failed: client connection to 127.5.107.66:35445: connect: Connection refused (error 111)
I20240502 14:26:18.230799 13456 leader_election.cc:302] T 003642773a724534847246de230ea34b P 375c6d5bdf924866b69b0e6667732692 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 375c6d5bdf924866b69b0e6667732692; no voters: 36e66c7b1a03458b89a91c56a57df72a
I20240502 14:26:18.231022 18499 raft_consensus.cc:2726] T 003642773a724534847246de230ea34b P 375c6d5bdf924866b69b0e6667732692 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:26:18.235841 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 13301
W20240502 14:26:18.282095 17248 connection.cc:535] client connection to 127.5.107.69:33193 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:26:18.282474 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 13436
W20240502 14:26:18.282451 17248 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:26:18.284416 17248 consensus_peers.cc:485] T 877f199681534650bc1d247cf7608f5f P 0321170b0cf04eac8a1033185d970e12 -> Peer 723fb53f34c94c6aa5f7ee3a65711268 (127.5.107.69:33193): Couldn't send request to peer 723fb53f34c94c6aa5f7ee3a65711268. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.69:33193: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:18.288060 13564 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:26:18.569402 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 13436
I20240502 14:26:18.614790 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 13572
I20240502 14:26:18.620085 13697 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:26:18.741067 18291 raft_consensus.cc:482] T 64e9d77a5b9343489db94f4988180213 P 0321170b0cf04eac8a1033185d970e12 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 36e66c7b1a03458b89a91c56a57df72a)
I20240502 14:26:18.741313 18291 raft_consensus.cc:504] T 64e9d77a5b9343489db94f4988180213 P 0321170b0cf04eac8a1033185d970e12 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 1160 OBSOLETE_local: false peers { permanent_uuid: "36e66c7b1a03458b89a91c56a57df72a" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 35445 } } peers { permanent_uuid: "0321170b0cf04eac8a1033185d970e12" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35275 } }
I20240502 14:26:18.742131 18291 leader_election.cc:288] T 64e9d77a5b9343489db94f4988180213 P 0321170b0cf04eac8a1033185d970e12 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445)
W20240502 14:26:18.743541 17252 leader_election.cc:334] T 64e9d77a5b9343489db94f4988180213 P 0321170b0cf04eac8a1033185d970e12 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 36e66c7b1a03458b89a91c56a57df72a (127.5.107.66:35445): Network error: Client connection negotiation failed: client connection to 127.5.107.66:35445: connect: Connection refused (error 111)
I20240502 14:26:18.743701 17252 leader_election.cc:302] T 64e9d77a5b9343489db94f4988180213 P 0321170b0cf04eac8a1033185d970e12 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0321170b0cf04eac8a1033185d970e12; no voters: 36e66c7b1a03458b89a91c56a57df72a
I20240502 14:26:18.743973 18291 raft_consensus.cc:2726] T 64e9d77a5b9343489db94f4988180213 P 0321170b0cf04eac8a1033185d970e12 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:26:18.879796 18502 raft_consensus.cc:482] T e05a3a4fb76d4c46908945ad0d838509 P b0d1c78721a746e99d0d3056f1e31371 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d9748c575342401bb80683c7e2c4e105)
I20240502 14:26:18.880301 18502 raft_consensus.cc:504] T e05a3a4fb76d4c46908945ad0d838509 P b0d1c78721a746e99d0d3056f1e31371 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "d9748c575342401bb80683c7e2c4e105" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45923 } } peers { permanent_uuid: "b0d1c78721a746e99d0d3056f1e31371" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 33799 } }
I20240502 14:26:18.881457 18502 leader_election.cc:288] T e05a3a4fb76d4c46908945ad0d838509 P b0d1c78721a746e99d0d3056f1e31371 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d9748c575342401bb80683c7e2c4e105 (127.5.107.67:45923)
I20240502 14:26:18.881686 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 13572
I20240502 14:26:18.927587 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 17233
I20240502 14:26:18.933126 17358 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:26:19.128866 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 17233
I20240502 14:26:19.128813 18291 raft_consensus.cc:482] T 1427194e32d545f785df3e45137aa42e P 0321170b0cf04eac8a1033185d970e12 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 723fb53f34c94c6aa5f7ee3a65711268)
I20240502 14:26:19.160892 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 11762
I20240502 14:26:19.161877 11823 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:26:19.410246 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 11762
2024-05-02T14:26:19Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/2 (111000 ms)
[----------] 1 test from RebalanceParamTest (111000 ms total)
[----------] 1 test from ConcurrentRebalancersTest
[ RUN ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0
I20240502 14:26:19.520932 5549 test_util.cc:268] Using random seed: 1660775315
I20240502 14:26:19.522715 5549 ts_itest-base.cc:112] Starting cluster with:
I20240502 14:26:19.522783 5549 ts_itest-base.cc:113] --------------
I20240502 14:26:19.522819 5549 ts_itest-base.cc:114] 8 tablet servers
I20240502 14:26:19.522861 5549 ts_itest-base.cc:115] 3 replicas per TS
I20240502 14:26:19.522902 5549 ts_itest-base.cc:116] --------------
2024-05-02T14:26:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:26:19Z Disabled control of system clock
I20240502 14:26:19.574854 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.107.126:33551
--webserver_interface=127.5.107.126
--webserver_port=0
--builtin_ntp_servers=127.5.107.84:43287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.107.126:33551
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240502 14:26:19.888662 18528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:26:19.888947 18528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:26:19.889088 18528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:26:19.895969 18528 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:26:19.896102 18528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:26:19.896185 18528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:26:19.896276 18528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:26:19.896342 18528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:26:19.904604 18528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:43287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.5.107.126:33551
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.107.126:33551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.107.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:26:19.905860 18528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:26:19.907552 18528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:26:19.917860 18534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:26:19.918526 18533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:26:19.919060 18536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:26:19.919746 18528 server_base.cc:1034] running on GCE node
I20240502 14:26:19.920326 18528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:26:19.921728 18528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:26:19.922920 18528 hybrid_clock.cc:648] HybridClock initialized: now 1714659979922876 us; error 42 us; skew 500 ppm
I20240502 14:26:19.923209 18528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:26:19.925304 18528 webserver.cc:460] Webserver started at http://127.5.107.126:35521/ using document root <none> and password file <none>
I20240502 14:26:19.926088 18528 fs_manager.cc:359] Metadata directory not provided
I20240502 14:26:19.926208 18528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:26:19.926512 18528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:26:19.928725 18528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0283476f38234a96bfe72594576461be"
format_stamp: "Formatted at 2024-05-02 14:26:19 on dist-test-slave-2cw1"
I20240502 14:26:19.929487 18528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0283476f38234a96bfe72594576461be"
format_stamp: "Formatted at 2024-05-02 14:26:19 on dist-test-slave-2cw1"
I20240502 14:26:19.933993 18528 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 14:26:19.936745 18542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:19.937861 18528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:26:19.938000 18528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0283476f38234a96bfe72594576461be"
format_stamp: "Formatted at 2024-05-02 14:26:19 on dist-test-slave-2cw1"
I20240502 14:26:19.938171 18528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:26:19.951187 18528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:26:19.951918 18528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:26:19.952164 18528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:26:19.968271 18528 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.126:33551
I20240502 14:26:19.968288 18594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.126:33551 every 8 connection(s)
I20240502 14:26:19.969424 18528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 14:26:19.970292 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 18528
I20240502 14:26:19.970507 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 14:26:19.973691 18595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:19.984308 18595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0283476f38234a96bfe72594576461be: Bootstrap starting.
I20240502 14:26:19.986748 18595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0283476f38234a96bfe72594576461be: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:19.987589 18595 log.cc:824] T 00000000000000000000000000000000 P 0283476f38234a96bfe72594576461be: Log is configured to *not* fsync() on all Append() calls
I20240502 14:26:19.989882 18595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0283476f38234a96bfe72594576461be: No bootstrap required, opened a new log
I20240502 14:26:19.995383 18595 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0283476f38234a96bfe72594576461be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0283476f38234a96bfe72594576461be" member_type: VOTER last_known_addr { host: "127.5.107.126" port: 33551 } }
I20240502 14:26:19.995702 18595 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0283476f38234a96bfe72594576461be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:19.995802 18595 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0283476f38234a96bfe72594576461be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0283476f38234a96bfe72594576461be, State: Initialized, Role: FOLLOWER
I20240502 14:26:19.996402 18595 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0283476f38234a96bfe72594576461be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0283476f38234a96bfe72594576461be" member_type: VOTER last_known_addr { host: "127.5.107.126" port: 33551 } }
I20240502 14:26:19.996596 18595 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0283476f38234a96bfe72594576461be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:26:19.996711 18595 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0283476f38234a96bfe72594576461be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:26:19.996845 18595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0283476f38234a96bfe72594576461be [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:19.998674 18595 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0283476f38234a96bfe72594576461be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0283476f38234a96bfe72594576461be" member_type: VOTER last_known_addr { host: "127.5.107.126" port: 33551 } }
I20240502 14:26:19.999102 18595 leader_election.cc:302] T 00000000000000000000000000000000 P 0283476f38234a96bfe72594576461be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0283476f38234a96bfe72594576461be; no voters:
I20240502 14:26:19.999563 18595 leader_election.cc:288] T 00000000000000000000000000000000 P 0283476f38234a96bfe72594576461be [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:26:19.999723 18600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0283476f38234a96bfe72594576461be [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:26:20.000284 18600 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0283476f38234a96bfe72594576461be [term 1 LEADER]: Becoming Leader. State: Replica: 0283476f38234a96bfe72594576461be, State: Running, Role: LEADER
I20240502 14:26:20.000799 18600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0283476f38234a96bfe72594576461be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0283476f38234a96bfe72594576461be" member_type: VOTER last_known_addr { host: "127.5.107.126" port: 33551 } }
I20240502 14:26:20.001235 18595 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0283476f38234a96bfe72594576461be [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:26:20.003616 18602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0283476f38234a96bfe72594576461be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0283476f38234a96bfe72594576461be. Latest consensus state: current_term: 1 leader_uuid: "0283476f38234a96bfe72594576461be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0283476f38234a96bfe72594576461be" member_type: VOTER last_known_addr { host: "127.5.107.126" port: 33551 } } }
I20240502 14:26:20.003711 18601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0283476f38234a96bfe72594576461be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0283476f38234a96bfe72594576461be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0283476f38234a96bfe72594576461be" member_type: VOTER last_known_addr { host: "127.5.107.126" port: 33551 } } }
I20240502 14:26:20.004063 18601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0283476f38234a96bfe72594576461be [sys.catalog]: This master's current role is: LEADER
I20240502 14:26:20.004041 18602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0283476f38234a96bfe72594576461be [sys.catalog]: This master's current role is: LEADER
I20240502 14:26:20.005558 18606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:26:20.009972 18606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:26:20.018419 18606 catalog_manager.cc:1347] Generated new cluster ID: 47f54f736bc44e5d9541b8048145dd12
I20240502 14:26:20.018513 18606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:26:20.027259 18606 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:26:20.028220 18606 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:26:20.035876 18606 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 0283476f38234a96bfe72594576461be: Generated new TSK 0
I20240502 14:26:20.036602 18606 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:26:20.050482 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.65:0
--local_ip_for_outbound_sockets=127.5.107.65
--webserver_interface=127.5.107.65
--webserver_port=0
--tserver_master_addrs=127.5.107.126:33551
--builtin_ntp_servers=127.5.107.84:43287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 14:26:20.362978 18619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:26:20.363219 18619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:26:20.363294 18619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:26:20.369506 18619 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:26:20.369593 18619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:26:20.369711 18619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.65
I20240502 14:26:20.377677 18619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:43287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.107.65
--webserver_port=0
--tserver_master_addrs=127.5.107.126:33551
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.65
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:26:20.378677 18619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:26:20.380156 18619 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:26:20.391348 18624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:26:20.391952 18625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:26:20.392470 18627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:26:20.393103 18619 server_base.cc:1034] running on GCE node
I20240502 14:26:20.393589 18619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:26:20.394315 18619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:26:20.395517 18619 hybrid_clock.cc:648] HybridClock initialized: now 1714659980395480 us; error 24 us; skew 500 ppm
I20240502 14:26:20.395797 18619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:26:20.397729 18619 webserver.cc:460] Webserver started at http://127.5.107.65:36589/ using document root <none> and password file <none>
I20240502 14:26:20.398350 18619 fs_manager.cc:359] Metadata directory not provided
I20240502 14:26:20.398463 18619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:26:20.398761 18619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:26:20.400980 18619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2db304b538e14857b7b1cc272d0db94b"
format_stamp: "Formatted at 2024-05-02 14:26:20 on dist-test-slave-2cw1"
I20240502 14:26:20.401782 18619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2db304b538e14857b7b1cc272d0db94b"
format_stamp: "Formatted at 2024-05-02 14:26:20 on dist-test-slave-2cw1"
I20240502 14:26:20.406170 18619 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 14:26:20.408975 18633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:20.410185 18619 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20240502 14:26:20.410344 18619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2db304b538e14857b7b1cc272d0db94b"
format_stamp: "Formatted at 2024-05-02 14:26:20 on dist-test-slave-2cw1"
I20240502 14:26:20.410519 18619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:26:20.433950 18619 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:26:20.434615 18619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:26:20.434859 18619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:26:20.435699 18619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:26:20.436944 18619 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:26:20.437033 18619 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:20.437136 18619 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:26:20.437216 18619 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:20.466687 18619 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.65:36749
I20240502 14:26:20.466749 18746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.65:36749 every 8 connection(s)
I20240502 14:26:20.467801 18619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 14:26:20.473497 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 18619
I20240502 14:26:20.473701 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 14:26:20.477075 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.66:0
--local_ip_for_outbound_sockets=127.5.107.66
--webserver_interface=127.5.107.66
--webserver_port=0
--tserver_master_addrs=127.5.107.126:33551
--builtin_ntp_servers=127.5.107.84:43287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:26:20.479141 18747 heartbeater.cc:347] Connected to a master server at 127.5.107.126:33551
I20240502 14:26:20.479365 18747 heartbeater.cc:464] Registering TS with master...
I20240502 14:26:20.479946 18747 heartbeater.cc:511] Master 127.5.107.126:33551 requested a full tablet report, sending...
I20240502 14:26:20.481832 18560 ts_manager.cc:194] Registered new tserver with Master: 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:26:20.483597 18560 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.65:58683
W20240502 14:26:20.795055 18750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:26:20.795293 18750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:26:20.795408 18750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:26:20.802420 18750 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:26:20.802523 18750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:26:20.802685 18750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.66
I20240502 14:26:20.811311 18750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:43287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.107.66
--webserver_port=0
--tserver_master_addrs=127.5.107.126:33551
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.66
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:26:20.812543 18750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:26:20.814244 18750 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:26:20.826268 18755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:26:20.826833 18756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:26:20.827304 18758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:26:20.828218 18750 server_base.cc:1034] running on GCE node
I20240502 14:26:20.828786 18750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:26:20.829746 18750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:26:20.830945 18750 hybrid_clock.cc:648] HybridClock initialized: now 1714659980830909 us; error 28 us; skew 500 ppm
I20240502 14:26:20.831235 18750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:26:20.833343 18750 webserver.cc:460] Webserver started at http://127.5.107.66:44201/ using document root <none> and password file <none>
I20240502 14:26:20.834007 18750 fs_manager.cc:359] Metadata directory not provided
I20240502 14:26:20.834110 18750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:26:20.834406 18750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:26:20.836694 18750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "fec04c50dc5b49878ffa3c3873df32e4"
format_stamp: "Formatted at 2024-05-02 14:26:20 on dist-test-slave-2cw1"
I20240502 14:26:20.837476 18750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "fec04c50dc5b49878ffa3c3873df32e4"
format_stamp: "Formatted at 2024-05-02 14:26:20 on dist-test-slave-2cw1"
I20240502 14:26:20.842197 18750 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 14:26:20.845690 18764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:20.846928 18750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20240502 14:26:20.847071 18750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "fec04c50dc5b49878ffa3c3873df32e4"
format_stamp: "Formatted at 2024-05-02 14:26:20 on dist-test-slave-2cw1"
I20240502 14:26:20.847239 18750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:26:20.864169 18750 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:26:20.864820 18750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:26:20.865046 18750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:26:20.865882 18750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:26:20.867156 18750 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:26:20.867256 18750 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:20.867372 18750 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:26:20.867447 18750 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:20.897162 18750 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.66:33009
I20240502 14:26:20.897238 18878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.66:33009 every 8 connection(s)
I20240502 14:26:20.898403 18750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 14:26:20.900151 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 18750
I20240502 14:26:20.900350 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 14:26:20.904162 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.67:0
--local_ip_for_outbound_sockets=127.5.107.67
--webserver_interface=127.5.107.67
--webserver_port=0
--tserver_master_addrs=127.5.107.126:33551
--builtin_ntp_servers=127.5.107.84:43287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:26:20.911432 18879 heartbeater.cc:347] Connected to a master server at 127.5.107.126:33551
I20240502 14:26:20.911759 18879 heartbeater.cc:464] Registering TS with master...
I20240502 14:26:20.912596 18879 heartbeater.cc:511] Master 127.5.107.126:33551 requested a full tablet report, sending...
I20240502 14:26:20.913967 18560 ts_manager.cc:194] Registered new tserver with Master: fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
I20240502 14:26:20.914705 18560 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.66:58595
W20240502 14:26:21.222936 18882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:26:21.223170 18882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:26:21.223248 18882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:26:21.229470 18882 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:26:21.229570 18882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:26:21.229676 18882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.67
I20240502 14:26:21.237174 18882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:43287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.107.67
--webserver_port=0
--tserver_master_addrs=127.5.107.126:33551
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.67
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:26:21.238260 18882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:26:21.239765 18882 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:26:21.250921 18890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:26:21.251554 18887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:26:21.252104 18888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:26:21.252817 18882 server_base.cc:1034] running on GCE node
I20240502 14:26:21.253324 18882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:26:21.254074 18882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:26:21.255242 18882 hybrid_clock.cc:648] HybridClock initialized: now 1714659981255198 us; error 41 us; skew 500 ppm
I20240502 14:26:21.255484 18882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:26:21.257287 18882 webserver.cc:460] Webserver started at http://127.5.107.67:46543/ using document root <none> and password file <none>
I20240502 14:26:21.257896 18882 fs_manager.cc:359] Metadata directory not provided
I20240502 14:26:21.257984 18882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:26:21.258257 18882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:26:21.260387 18882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "c902805060df4df9b5599d0e94b2dc06"
format_stamp: "Formatted at 2024-05-02 14:26:21 on dist-test-slave-2cw1"
I20240502 14:26:21.261085 18882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "c902805060df4df9b5599d0e94b2dc06"
format_stamp: "Formatted at 2024-05-02 14:26:21 on dist-test-slave-2cw1"
I20240502 14:26:21.265408 18882 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 14:26:21.268100 18896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:21.269227 18882 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:26:21.269376 18882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "c902805060df4df9b5599d0e94b2dc06"
format_stamp: "Formatted at 2024-05-02 14:26:21 on dist-test-slave-2cw1"
I20240502 14:26:21.269533 18882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:26:21.281831 18882 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:26:21.282502 18882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:26:21.282759 18882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:26:21.283586 18882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:26:21.284847 18882 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:26:21.284935 18882 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:21.285038 18882 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:26:21.285106 18882 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:21.314167 18882 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.67:45409
I20240502 14:26:21.314213 19010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.67:45409 every 8 connection(s)
I20240502 14:26:21.315321 18882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 14:26:21.320156 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 18882
I20240502 14:26:21.320331 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 14:26:21.323757 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.68:0
--local_ip_for_outbound_sockets=127.5.107.68
--webserver_interface=127.5.107.68
--webserver_port=0
--tserver_master_addrs=127.5.107.126:33551
--builtin_ntp_servers=127.5.107.84:43287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:26:21.326977 19011 heartbeater.cc:347] Connected to a master server at 127.5.107.126:33551
I20240502 14:26:21.327214 19011 heartbeater.cc:464] Registering TS with master...
I20240502 14:26:21.327839 19011 heartbeater.cc:511] Master 127.5.107.126:33551 requested a full tablet report, sending...
I20240502 14:26:21.328998 18560 ts_manager.cc:194] Registered new tserver with Master: c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:26:21.329710 18560 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.67:59943
I20240502 14:26:21.486433 18747 heartbeater.cc:503] Master 127.5.107.126:33551 was elected leader, sending a full tablet report...
W20240502 14:26:21.637198 19014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:26:21.637456 19014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:26:21.637557 19014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:26:21.644306 19014 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:26:21.644402 19014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:26:21.644555 19014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.68
I20240502 14:26:21.652518 19014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:43287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.5.107.68
--webserver_port=0
--tserver_master_addrs=127.5.107.126:33551
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.68
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:26:21.653682 19014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:26:21.655170 19014 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:26:21.666064 19020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:26:21.666651 19019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:26:21.667131 19022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:26:21.667876 19014 server_base.cc:1034] running on GCE node
I20240502 14:26:21.668422 19014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:26:21.669174 19014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:26:21.670392 19014 hybrid_clock.cc:648] HybridClock initialized: now 1714659981670346 us; error 40 us; skew 500 ppm
I20240502 14:26:21.670645 19014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:26:21.678632 19014 webserver.cc:460] Webserver started at http://127.5.107.68:41507/ using document root <none> and password file <none>
I20240502 14:26:21.679260 19014 fs_manager.cc:359] Metadata directory not provided
I20240502 14:26:21.679376 19014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:26:21.679708 19014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:26:21.682121 19014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "007b2e79dc424a6f80141180f072f2d8"
format_stamp: "Formatted at 2024-05-02 14:26:21 on dist-test-slave-2cw1"
I20240502 14:26:21.682834 19014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "007b2e79dc424a6f80141180f072f2d8"
format_stamp: "Formatted at 2024-05-02 14:26:21 on dist-test-slave-2cw1"
I20240502 14:26:21.687053 19014 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:26:21.689837 19028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:21.690907 19014 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240502 14:26:21.691044 19014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "007b2e79dc424a6f80141180f072f2d8"
format_stamp: "Formatted at 2024-05-02 14:26:21 on dist-test-slave-2cw1"
I20240502 14:26:21.691205 19014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:26:21.709453 19014 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:26:21.710109 19014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:26:21.710333 19014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:26:21.711140 19014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:26:21.712426 19014 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:26:21.712522 19014 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:21.712638 19014 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:26:21.712710 19014 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:21.741714 19014 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.68:41463
I20240502 14:26:21.741745 19141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.68:41463 every 8 connection(s)
I20240502 14:26:21.742976 19014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:26:21.747655 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 19014
I20240502 14:26:21.747834 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 14:26:21.750808 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.69:0
--local_ip_for_outbound_sockets=127.5.107.69
--webserver_interface=127.5.107.69
--webserver_port=0
--tserver_master_addrs=127.5.107.126:33551
--builtin_ntp_servers=127.5.107.84:43287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:26:21.755005 19142 heartbeater.cc:347] Connected to a master server at 127.5.107.126:33551
I20240502 14:26:21.755261 19142 heartbeater.cc:464] Registering TS with master...
I20240502 14:26:21.755987 19142 heartbeater.cc:511] Master 127.5.107.126:33551 requested a full tablet report, sending...
I20240502 14:26:21.757280 18560 ts_manager.cc:194] Registered new tserver with Master: 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:26:21.758276 18560 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.68:43283
I20240502 14:26:21.917243 18879 heartbeater.cc:503] Master 127.5.107.126:33551 was elected leader, sending a full tablet report...
W20240502 14:26:22.066025 19145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:26:22.066247 19145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:26:22.066367 19145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:26:22.072777 19145 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:26:22.072871 19145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:26:22.073065 19145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.69
I20240502 14:26:22.080320 19145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:43287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.5.107.69
--webserver_port=0
--tserver_master_addrs=127.5.107.126:33551
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.69
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:26:22.081462 19145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:26:22.083125 19145 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:26:22.093508 19151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:26:22.094080 19150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:26:22.094528 19153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:26:22.095108 19145 server_base.cc:1034] running on GCE node
I20240502 14:26:22.095655 19145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:26:22.096415 19145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:26:22.097584 19145 hybrid_clock.cc:648] HybridClock initialized: now 1714659982097549 us; error 29 us; skew 500 ppm
I20240502 14:26:22.097836 19145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:26:22.099601 19145 webserver.cc:460] Webserver started at http://127.5.107.69:46705/ using document root <none> and password file <none>
I20240502 14:26:22.100168 19145 fs_manager.cc:359] Metadata directory not provided
I20240502 14:26:22.100256 19145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:26:22.100564 19145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:26:22.102946 19145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "a834edffbcb2469aae598b069aab0681"
format_stamp: "Formatted at 2024-05-02 14:26:22 on dist-test-slave-2cw1"
I20240502 14:26:22.103646 19145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "a834edffbcb2469aae598b069aab0681"
format_stamp: "Formatted at 2024-05-02 14:26:22 on dist-test-slave-2cw1"
I20240502 14:26:22.107834 19145 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 14:26:22.110653 19159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:22.111768 19145 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:26:22.111936 19145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "a834edffbcb2469aae598b069aab0681"
format_stamp: "Formatted at 2024-05-02 14:26:22 on dist-test-slave-2cw1"
I20240502 14:26:22.112109 19145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:26:22.133934 19145 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:26:22.134639 19145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:26:22.134886 19145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:26:22.135677 19145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:26:22.136953 19145 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:26:22.137040 19145 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:22.137143 19145 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:26:22.137238 19145 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:22.165784 19145 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.69:41119
I20240502 14:26:22.165841 19272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.69:41119 every 8 connection(s)
I20240502 14:26:22.167215 19145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:26:22.175120 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 19145
I20240502 14:26:22.175308 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 14:26:22.178485 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.70:0
--local_ip_for_outbound_sockets=127.5.107.70
--webserver_interface=127.5.107.70
--webserver_port=0
--tserver_master_addrs=127.5.107.126:33551
--builtin_ntp_servers=127.5.107.84:43287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:26:22.179075 19273 heartbeater.cc:347] Connected to a master server at 127.5.107.126:33551
I20240502 14:26:22.179322 19273 heartbeater.cc:464] Registering TS with master...
I20240502 14:26:22.179972 19273 heartbeater.cc:511] Master 127.5.107.126:33551 requested a full tablet report, sending...
I20240502 14:26:22.181308 18560 ts_manager.cc:194] Registered new tserver with Master: a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119)
I20240502 14:26:22.182374 18560 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.69:45143
I20240502 14:26:22.331876 19011 heartbeater.cc:503] Master 127.5.107.126:33551 was elected leader, sending a full tablet report...
W20240502 14:26:22.496115 19276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:26:22.496338 19276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:26:22.496455 19276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:26:22.503235 19276 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:26:22.503322 19276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:26:22.503474 19276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.70
I20240502 14:26:22.510913 19276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:43287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.5.107.70
--webserver_port=0
--tserver_master_addrs=127.5.107.126:33551
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.70
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:26:22.512028 19276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:26:22.513687 19276 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:26:22.525017 19281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:26:22.525568 19282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:26:22.526093 19284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:26:22.526759 19276 server_base.cc:1034] running on GCE node
I20240502 14:26:22.527253 19276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:26:22.528072 19276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:26:22.529271 19276 hybrid_clock.cc:648] HybridClock initialized: now 1714659982529226 us; error 33 us; skew 500 ppm
I20240502 14:26:22.529551 19276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:26:22.531414 19276 webserver.cc:460] Webserver started at http://127.5.107.70:32901/ using document root <none> and password file <none>
I20240502 14:26:22.532074 19276 fs_manager.cc:359] Metadata directory not provided
I20240502 14:26:22.532192 19276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:26:22.532497 19276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:26:22.534804 19276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0"
format_stamp: "Formatted at 2024-05-02 14:26:22 on dist-test-slave-2cw1"
I20240502 14:26:22.535522 19276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0"
format_stamp: "Formatted at 2024-05-02 14:26:22 on dist-test-slave-2cw1"
I20240502 14:26:22.539906 19276 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 14:26:22.543037 19290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:22.544159 19276 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:26:22.544314 19276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0"
format_stamp: "Formatted at 2024-05-02 14:26:22 on dist-test-slave-2cw1"
I20240502 14:26:22.544484 19276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:26:22.566577 19276 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:26:22.567333 19276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:26:22.567591 19276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:26:22.568432 19276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:26:22.569836 19276 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:26:22.569931 19276 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:22.570040 19276 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:26:22.570111 19276 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:22.599589 19276 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.70:46021
I20240502 14:26:22.599632 19403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.70:46021 every 8 connection(s)
I20240502 14:26:22.600795 19276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 14:26:22.602892 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 19276
I20240502 14:26:22.603107 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 14:26:22.606209 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.71:0
--local_ip_for_outbound_sockets=127.5.107.71
--webserver_interface=127.5.107.71
--webserver_port=0
--tserver_master_addrs=127.5.107.126:33551
--builtin_ntp_servers=127.5.107.84:43287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:26:22.612670 19404 heartbeater.cc:347] Connected to a master server at 127.5.107.126:33551
I20240502 14:26:22.612982 19404 heartbeater.cc:464] Registering TS with master...
I20240502 14:26:22.613831 19404 heartbeater.cc:511] Master 127.5.107.126:33551 requested a full tablet report, sending...
I20240502 14:26:22.615136 18560 ts_manager.cc:194] Registered new tserver with Master: a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021)
I20240502 14:26:22.615892 18560 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.70:52797
I20240502 14:26:22.760838 19142 heartbeater.cc:503] Master 127.5.107.126:33551 was elected leader, sending a full tablet report...
W20240502 14:26:22.924077 19407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:26:22.924332 19407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:26:22.924419 19407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:26:22.930671 19407 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:26:22.930758 19407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:26:22.930850 19407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.71
I20240502 14:26:22.938277 19407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:43287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.5.107.71
--webserver_port=0
--tserver_master_addrs=127.5.107.126:33551
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.71
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:26:22.939250 19407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:26:22.940719 19407 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:26:22.951694 19415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:26:22.952318 19412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:26:22.952783 19413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:26:22.953431 19407 server_base.cc:1034] running on GCE node
I20240502 14:26:22.953893 19407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:26:22.954602 19407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:26:22.955793 19407 hybrid_clock.cc:648] HybridClock initialized: now 1714659982955760 us; error 23 us; skew 500 ppm
I20240502 14:26:22.956053 19407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:26:22.957907 19407 webserver.cc:460] Webserver started at http://127.5.107.71:46407/ using document root <none> and password file <none>
I20240502 14:26:22.958529 19407 fs_manager.cc:359] Metadata directory not provided
I20240502 14:26:22.958611 19407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:26:22.958870 19407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:26:22.960959 19407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "7ab6d395b30d4afb9821b61c9454430b"
format_stamp: "Formatted at 2024-05-02 14:26:22 on dist-test-slave-2cw1"
I20240502 14:26:22.961645 19407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "7ab6d395b30d4afb9821b61c9454430b"
format_stamp: "Formatted at 2024-05-02 14:26:22 on dist-test-slave-2cw1"
I20240502 14:26:22.966029 19407 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 14:26:22.968804 19421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:22.970077 19407 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:26:22.970221 19407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "7ab6d395b30d4afb9821b61c9454430b"
format_stamp: "Formatted at 2024-05-02 14:26:22 on dist-test-slave-2cw1"
I20240502 14:26:22.970410 19407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:26:22.997714 19407 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:26:22.998337 19407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:26:22.998548 19407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:26:22.999313 19407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:26:23.000551 19407 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:26:23.000631 19407 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:23.000708 19407 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:26:23.000761 19407 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:23.030509 19407 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.71:39195
I20240502 14:26:23.030547 19534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.71:39195 every 8 connection(s)
I20240502 14:26:23.031685 19407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 14:26:23.032982 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 19407
I20240502 14:26:23.033238 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 14:26:23.036408 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.72:0
--local_ip_for_outbound_sockets=127.5.107.72
--webserver_interface=127.5.107.72
--webserver_port=0
--tserver_master_addrs=127.5.107.126:33551
--builtin_ntp_servers=127.5.107.84:43287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:26:23.044198 19535 heartbeater.cc:347] Connected to a master server at 127.5.107.126:33551
I20240502 14:26:23.044435 19535 heartbeater.cc:464] Registering TS with master...
I20240502 14:26:23.045059 19535 heartbeater.cc:511] Master 127.5.107.126:33551 requested a full tablet report, sending...
I20240502 14:26:23.046386 18560 ts_manager.cc:194] Registered new tserver with Master: 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
I20240502 14:26:23.047473 18560 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.71:43205
I20240502 14:26:23.184708 19273 heartbeater.cc:503] Master 127.5.107.126:33551 was elected leader, sending a full tablet report...
W20240502 14:26:23.354849 19539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:26:23.355104 19539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:26:23.355183 19539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:26:23.361663 19539 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:26:23.361755 19539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:26:23.361864 19539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.72
I20240502 14:26:23.369541 19539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:43287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.5.107.72
--webserver_port=0
--tserver_master_addrs=127.5.107.126:33551
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.72
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:26:23.370564 19539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:26:23.372126 19539 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:26:23.383512 19544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:26:23.384080 19545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:26:23.384680 19547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:26:23.385264 19539 server_base.cc:1034] running on GCE node
I20240502 14:26:23.385744 19539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:26:23.386469 19539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:26:23.387643 19539 hybrid_clock.cc:648] HybridClock initialized: now 1714659983387613 us; error 24 us; skew 500 ppm
I20240502 14:26:23.387876 19539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:26:23.389812 19539 webserver.cc:460] Webserver started at http://127.5.107.72:36821/ using document root <none> and password file <none>
I20240502 14:26:23.390389 19539 fs_manager.cc:359] Metadata directory not provided
I20240502 14:26:23.390470 19539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:26:23.390726 19539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:26:23.393131 19539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "ff6acba046404ef0a6f41ad5e09d852f"
format_stamp: "Formatted at 2024-05-02 14:26:23 on dist-test-slave-2cw1"
I20240502 14:26:23.393810 19539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "ff6acba046404ef0a6f41ad5e09d852f"
format_stamp: "Formatted at 2024-05-02 14:26:23 on dist-test-slave-2cw1"
I20240502 14:26:23.398079 19539 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 14:26:23.400841 19553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:23.402134 19539 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20240502 14:26:23.402259 19539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "ff6acba046404ef0a6f41ad5e09d852f"
format_stamp: "Formatted at 2024-05-02 14:26:23 on dist-test-slave-2cw1"
I20240502 14:26:23.402415 19539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:26:23.416031 19539 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:26:23.416661 19539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:26:23.416869 19539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:26:23.417642 19539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:26:23.418776 19539 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:26:23.418852 19539 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:23.418922 19539 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:26:23.418974 19539 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:23.447085 19539 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.72:35871
I20240502 14:26:23.447139 19666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.72:35871 every 8 connection(s)
I20240502 14:26:23.448220 19539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 14:26:23.452036 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 19539
I20240502 14:26:23.452229 5549 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 14:26:23.459625 19667 heartbeater.cc:347] Connected to a master server at 127.5.107.126:33551
I20240502 14:26:23.459838 19667 heartbeater.cc:464] Registering TS with master...
I20240502 14:26:23.460399 19667 heartbeater.cc:511] Master 127.5.107.126:33551 requested a full tablet report, sending...
I20240502 14:26:23.461551 18560 ts_manager.cc:194] Registered new tserver with Master: ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871)
I20240502 14:26:23.462333 18560 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.72:44187
I20240502 14:26:23.467628 5549 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 14:26:23.482342 5549 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 14:26:23.482450 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 19145
I20240502 14:26:23.489040 19268 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:26:23.564251 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 19145
I20240502 14:26:23.581509 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 19276
I20240502 14:26:23.589085 19399 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:26:23.665493 19404 heartbeater.cc:503] Master 127.5.107.126:33551 was elected leader, sending a full tablet report...
I20240502 14:26:23.666718 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 19276
W20240502 14:26:23.681565 18545 connection.cc:535] server connection from 127.5.107.70:52797 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:26:23.681843 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 19407
I20240502 14:26:23.689440 19530 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:26:23.761709 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 19407
I20240502 14:26:23.777791 5549 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu pid 19539
I20240502 14:26:23.785302 19662 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:26:23.863178 5549 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu with pid 19539
I20240502 14:26:27.632061 18560 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48228:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:26:27.687636 19077 tablet_service.cc:1467] Processing CreateTablet for tablet d28db16c9f7e405b8866f0e6c548e246 (DEFAULT_TABLE table=rebalance_test_table_0 [id=27066b01c29c4a1380ee4753a1c036d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:27.687647 19076 tablet_service.cc:1467] Processing CreateTablet for tablet cf2a6426ae6c438f942d39530da6e3e3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=27066b01c29c4a1380ee4753a1c036d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:27.689384 19077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d28db16c9f7e405b8866f0e6c548e246. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:27.692093 18944 tablet_service.cc:1467] Processing CreateTablet for tablet cf2a6426ae6c438f942d39530da6e3e3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=27066b01c29c4a1380ee4753a1c036d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:27.692800 18813 tablet_service.cc:1467] Processing CreateTablet for tablet d28db16c9f7e405b8866f0e6c548e246 (DEFAULT_TABLE table=rebalance_test_table_0 [id=27066b01c29c4a1380ee4753a1c036d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:27.693760 18944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf2a6426ae6c438f942d39530da6e3e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:27.694242 18813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d28db16c9f7e405b8866f0e6c548e246. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:27.694509 19076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf2a6426ae6c438f942d39530da6e3e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:27.692065 18945 tablet_service.cc:1467] Processing CreateTablet for tablet 9580f3fdf7084cc0923b1b16bfedd450 (DEFAULT_TABLE table=rebalance_test_table_0 [id=27066b01c29c4a1380ee4753a1c036d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:27.694968 18812 tablet_service.cc:1467] Processing CreateTablet for tablet 9580f3fdf7084cc0923b1b16bfedd450 (DEFAULT_TABLE table=rebalance_test_table_0 [id=27066b01c29c4a1380ee4753a1c036d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:27.695149 18945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9580f3fdf7084cc0923b1b16bfedd450. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:27.695413 18812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9580f3fdf7084cc0923b1b16bfedd450. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:27.703408 18679 tablet_service.cc:1467] Processing CreateTablet for tablet cf2a6426ae6c438f942d39530da6e3e3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=27066b01c29c4a1380ee4753a1c036d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:27.703366 18680 tablet_service.cc:1467] Processing CreateTablet for tablet 9580f3fdf7084cc0923b1b16bfedd450 (DEFAULT_TABLE table=rebalance_test_table_0 [id=27066b01c29c4a1380ee4753a1c036d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:27.700449 18681 tablet_service.cc:1467] Processing CreateTablet for tablet d28db16c9f7e405b8866f0e6c548e246 (DEFAULT_TABLE table=rebalance_test_table_0 [id=27066b01c29c4a1380ee4753a1c036d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:27.704777 18680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9580f3fdf7084cc0923b1b16bfedd450. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:27.707603 18679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf2a6426ae6c438f942d39530da6e3e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:27.708832 18681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d28db16c9f7e405b8866f0e6c548e246. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:27.716528 19699 tablet_bootstrap.cc:492] T 9580f3fdf7084cc0923b1b16bfedd450 P c902805060df4df9b5599d0e94b2dc06: Bootstrap starting.
I20240502 14:26:27.720876 19700 tablet_bootstrap.cc:492] T d28db16c9f7e405b8866f0e6c548e246 P fec04c50dc5b49878ffa3c3873df32e4: Bootstrap starting.
I20240502 14:26:27.723318 19699 tablet_bootstrap.cc:654] T 9580f3fdf7084cc0923b1b16bfedd450 P c902805060df4df9b5599d0e94b2dc06: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:27.724733 19700 tablet_bootstrap.cc:654] T d28db16c9f7e405b8866f0e6c548e246 P fec04c50dc5b49878ffa3c3873df32e4: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:27.725405 19699 log.cc:824] T 9580f3fdf7084cc0923b1b16bfedd450 P c902805060df4df9b5599d0e94b2dc06: Log is configured to *not* fsync() on all Append() calls
I20240502 14:26:27.726001 19700 log.cc:824] T d28db16c9f7e405b8866f0e6c548e246 P fec04c50dc5b49878ffa3c3873df32e4: Log is configured to *not* fsync() on all Append() calls
I20240502 14:26:27.756660 19703 tablet_bootstrap.cc:492] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b: Bootstrap starting.
I20240502 14:26:27.763208 19698 tablet_bootstrap.cc:492] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8: Bootstrap starting.
I20240502 14:26:27.763710 19703 tablet_bootstrap.cc:654] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:27.765051 19699 tablet_bootstrap.cc:492] T 9580f3fdf7084cc0923b1b16bfedd450 P c902805060df4df9b5599d0e94b2dc06: No bootstrap required, opened a new log
I20240502 14:26:27.765415 19699 ts_tablet_manager.cc:1367] T 9580f3fdf7084cc0923b1b16bfedd450 P c902805060df4df9b5599d0e94b2dc06: Time spent bootstrapping tablet: real 0.049s user 0.016s sys 0.025s
I20240502 14:26:27.765549 19700 tablet_bootstrap.cc:492] T d28db16c9f7e405b8866f0e6c548e246 P fec04c50dc5b49878ffa3c3873df32e4: No bootstrap required, opened a new log
I20240502 14:26:27.765936 19700 ts_tablet_manager.cc:1367] T d28db16c9f7e405b8866f0e6c548e246 P fec04c50dc5b49878ffa3c3873df32e4: Time spent bootstrapping tablet: real 0.045s user 0.028s sys 0.015s
I20240502 14:26:27.766645 19698 tablet_bootstrap.cc:654] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:27.766709 19703 log.cc:824] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b: Log is configured to *not* fsync() on all Append() calls
I20240502 14:26:27.768110 19698 log.cc:824] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8: Log is configured to *not* fsync() on all Append() calls
I20240502 14:26:27.775358 19700 raft_consensus.cc:348] T d28db16c9f7e405b8866f0e6c548e246 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:27.775635 19699 raft_consensus.cc:348] T 9580f3fdf7084cc0923b1b16bfedd450 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:27.776105 19699 raft_consensus.cc:374] T 9580f3fdf7084cc0923b1b16bfedd450 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:27.776105 19700 raft_consensus.cc:374] T d28db16c9f7e405b8866f0e6c548e246 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:27.776239 19699 raft_consensus.cc:729] T 9580f3fdf7084cc0923b1b16bfedd450 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Initialized, Role: FOLLOWER
I20240502 14:26:27.776240 19700 raft_consensus.cc:729] T d28db16c9f7e405b8866f0e6c548e246 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Initialized, Role: FOLLOWER
I20240502 14:26:27.776991 19699 consensus_queue.cc:260] T 9580f3fdf7084cc0923b1b16bfedd450 P c902805060df4df9b5599d0e94b2dc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:27.777035 19700 consensus_queue.cc:260] T d28db16c9f7e405b8866f0e6c548e246 P fec04c50dc5b49878ffa3c3873df32e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:27.780267 19698 tablet_bootstrap.cc:492] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8: No bootstrap required, opened a new log
I20240502 14:26:27.780817 19698 ts_tablet_manager.cc:1367] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.000s
I20240502 14:26:27.787508 19699 ts_tablet_manager.cc:1398] T 9580f3fdf7084cc0923b1b16bfedd450 P c902805060df4df9b5599d0e94b2dc06: Time spent starting tablet: real 0.022s user 0.012s sys 0.007s
I20240502 14:26:27.788178 19703 tablet_bootstrap.cc:492] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b: No bootstrap required, opened a new log
I20240502 14:26:27.789619 19700 ts_tablet_manager.cc:1398] T d28db16c9f7e405b8866f0e6c548e246 P fec04c50dc5b49878ffa3c3873df32e4: Time spent starting tablet: real 0.023s user 0.014s sys 0.008s
I20240502 14:26:27.790148 19699 tablet_bootstrap.cc:492] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06: Bootstrap starting.
I20240502 14:26:27.790148 19700 tablet_bootstrap.cc:492] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4: Bootstrap starting.
I20240502 14:26:27.788581 19703 ts_tablet_manager.cc:1367] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.016s
I20240502 14:26:27.791944 19699 tablet_bootstrap.cc:654] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:27.794621 19700 tablet_bootstrap.cc:654] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:27.799963 19703 raft_consensus.cc:348] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:27.800644 19703 raft_consensus.cc:374] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:27.800801 19703 raft_consensus.cc:729] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Initialized, Role: FOLLOWER
I20240502 14:26:27.801817 19703 consensus_queue.cc:260] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:27.801800 19698 raft_consensus.cc:348] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:27.802326 19698 raft_consensus.cc:374] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:27.802461 19698 raft_consensus.cc:729] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Initialized, Role: FOLLOWER
I20240502 14:26:27.803220 19698 consensus_queue.cc:260] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:27.803481 19700 tablet_bootstrap.cc:492] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4: No bootstrap required, opened a new log
I20240502 14:26:27.803658 19700 ts_tablet_manager.cc:1367] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.005s
I20240502 14:26:27.804726 19700 raft_consensus.cc:348] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:27.804944 19700 raft_consensus.cc:374] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:27.805027 19700 raft_consensus.cc:729] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Initialized, Role: FOLLOWER
I20240502 14:26:27.805271 19700 consensus_queue.cc:260] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:27.805619 19699 tablet_bootstrap.cc:492] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06: No bootstrap required, opened a new log
I20240502 14:26:27.805796 19699 ts_tablet_manager.cc:1367] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20240502 14:26:27.805920 19700 ts_tablet_manager.cc:1398] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:26:27.805924 19703 ts_tablet_manager.cc:1398] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20240502 14:26:27.805831 19710 raft_consensus.cc:482] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:27.806949 19699 raft_consensus.cc:348] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:27.807278 19699 raft_consensus.cc:374] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:27.807379 19699 raft_consensus.cc:729] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Initialized, Role: FOLLOWER
I20240502 14:26:27.806520 19703 tablet_bootstrap.cc:492] T cf2a6426ae6c438f942d39530da6e3e3 P 2db304b538e14857b7b1cc272d0db94b: Bootstrap starting.
I20240502 14:26:27.807643 19699 consensus_queue.cc:260] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:27.808624 19699 ts_tablet_manager.cc:1398] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:26:27.809221 19710 raft_consensus.cc:504] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:27.810083 19703 tablet_bootstrap.cc:654] T cf2a6426ae6c438f942d39530da6e3e3 P 2db304b538e14857b7b1cc272d0db94b: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:27.815454 19710 leader_election.cc:288] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009), 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:26:27.817462 19698 ts_tablet_manager.cc:1398] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8: Time spent starting tablet: real 0.036s user 0.011s sys 0.013s
I20240502 14:26:27.818022 19698 tablet_bootstrap.cc:492] T cf2a6426ae6c438f942d39530da6e3e3 P 007b2e79dc424a6f80141180f072f2d8: Bootstrap starting.
I20240502 14:26:27.820384 19698 tablet_bootstrap.cc:654] T cf2a6426ae6c438f942d39530da6e3e3 P 007b2e79dc424a6f80141180f072f2d8: Neither blocks nor log segments found. Creating new log.
W20240502 14:26:27.821933 19012 tablet.cc:2369] T 9580f3fdf7084cc0923b1b16bfedd450 P c902805060df4df9b5599d0e94b2dc06: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:26:27.825531 18833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d28db16c9f7e405b8866f0e6c548e246" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec04c50dc5b49878ffa3c3873df32e4" is_pre_election: true
I20240502 14:26:27.826138 18833 raft_consensus.cc:2454] T d28db16c9f7e405b8866f0e6c548e246 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2db304b538e14857b7b1cc272d0db94b in term 0.
I20240502 14:26:27.826500 19703 tablet_bootstrap.cc:492] T cf2a6426ae6c438f942d39530da6e3e3 P 2db304b538e14857b7b1cc272d0db94b: No bootstrap required, opened a new log
I20240502 14:26:27.826690 19703 ts_tablet_manager.cc:1367] T cf2a6426ae6c438f942d39530da6e3e3 P 2db304b538e14857b7b1cc272d0db94b: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.009s
I20240502 14:26:27.826980 18634 leader_election.cc:302] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b, fec04c50dc5b49878ffa3c3873df32e4; no voters:
I20240502 14:26:27.827119 19698 tablet_bootstrap.cc:492] T cf2a6426ae6c438f942d39530da6e3e3 P 007b2e79dc424a6f80141180f072f2d8: No bootstrap required, opened a new log
I20240502 14:26:27.827401 19698 ts_tablet_manager.cc:1367] T cf2a6426ae6c438f942d39530da6e3e3 P 007b2e79dc424a6f80141180f072f2d8: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20240502 14:26:27.827389 19710 raft_consensus.cc:2781] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:27.827549 19710 raft_consensus.cc:482] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:27.827664 19710 raft_consensus.cc:3037] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:27.828022 19703 raft_consensus.cc:348] T cf2a6426ae6c438f942d39530da6e3e3 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:27.828241 19703 raft_consensus.cc:374] T cf2a6426ae6c438f942d39530da6e3e3 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:27.828346 19703 raft_consensus.cc:729] T cf2a6426ae6c438f942d39530da6e3e3 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Initialized, Role: FOLLOWER
I20240502 14:26:27.828699 19703 consensus_queue.cc:260] T cf2a6426ae6c438f942d39530da6e3e3 P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:27.828840 19698 raft_consensus.cc:348] T cf2a6426ae6c438f942d39530da6e3e3 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:27.829175 19698 raft_consensus.cc:374] T cf2a6426ae6c438f942d39530da6e3e3 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:27.829341 19698 raft_consensus.cc:729] T cf2a6426ae6c438f942d39530da6e3e3 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Initialized, Role: FOLLOWER
I20240502 14:26:27.829691 19698 consensus_queue.cc:260] T cf2a6426ae6c438f942d39530da6e3e3 P 007b2e79dc424a6f80141180f072f2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:27.830431 19698 ts_tablet_manager.cc:1398] T cf2a6426ae6c438f942d39530da6e3e3 P 007b2e79dc424a6f80141180f072f2d8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:26:27.830453 19097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d28db16c9f7e405b8866f0e6c548e246" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b2e79dc424a6f80141180f072f2d8" is_pre_election: true
I20240502 14:26:27.830617 19710 raft_consensus.cc:504] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:27.831030 19097 raft_consensus.cc:2454] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2db304b538e14857b7b1cc272d0db94b in term 0.
I20240502 14:26:27.831348 19710 leader_election.cc:288] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 election: Requested vote from peers fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009), 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:26:27.832509 19097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d28db16c9f7e405b8866f0e6c548e246" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b2e79dc424a6f80141180f072f2d8"
I20240502 14:26:27.832746 19097 raft_consensus.cc:3037] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:27.833709 19703 ts_tablet_manager.cc:1398] T cf2a6426ae6c438f942d39530da6e3e3 P 2db304b538e14857b7b1cc272d0db94b: Time spent starting tablet: real 0.007s user 0.001s sys 0.003s
I20240502 14:26:27.833681 18833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d28db16c9f7e405b8866f0e6c548e246" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec04c50dc5b49878ffa3c3873df32e4"
I20240502 14:26:27.833904 18833 raft_consensus.cc:3037] T d28db16c9f7e405b8866f0e6c548e246 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:27.834105 19703 tablet_bootstrap.cc:492] T 9580f3fdf7084cc0923b1b16bfedd450 P 2db304b538e14857b7b1cc272d0db94b: Bootstrap starting.
I20240502 14:26:27.835896 19703 tablet_bootstrap.cc:654] T 9580f3fdf7084cc0923b1b16bfedd450 P 2db304b538e14857b7b1cc272d0db94b: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:27.836449 19097 raft_consensus.cc:2454] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2db304b538e14857b7b1cc272d0db94b in term 1.
I20240502 14:26:27.836906 18637 leader_election.cc:302] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 007b2e79dc424a6f80141180f072f2d8, 2db304b538e14857b7b1cc272d0db94b; no voters:
I20240502 14:26:27.837028 18833 raft_consensus.cc:2454] T d28db16c9f7e405b8866f0e6c548e246 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2db304b538e14857b7b1cc272d0db94b in term 1.
I20240502 14:26:27.837451 19716 raft_consensus.cc:2781] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:26:27.837767 19716 raft_consensus.cc:686] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Becoming Leader. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Running, Role: LEADER
I20240502 14:26:27.838382 19716 consensus_queue.cc:237] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:27.839130 19703 tablet_bootstrap.cc:492] T 9580f3fdf7084cc0923b1b16bfedd450 P 2db304b538e14857b7b1cc272d0db94b: No bootstrap required, opened a new log
I20240502 14:26:27.839303 19703 ts_tablet_manager.cc:1367] T 9580f3fdf7084cc0923b1b16bfedd450 P 2db304b538e14857b7b1cc272d0db94b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 14:26:27.840015 19703 raft_consensus.cc:348] T 9580f3fdf7084cc0923b1b16bfedd450 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:27.840231 19703 raft_consensus.cc:374] T 9580f3fdf7084cc0923b1b16bfedd450 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:27.840308 19703 raft_consensus.cc:729] T 9580f3fdf7084cc0923b1b16bfedd450 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Initialized, Role: FOLLOWER
I20240502 14:26:27.840566 19703 consensus_queue.cc:260] T 9580f3fdf7084cc0923b1b16bfedd450 P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:27.841252 19703 ts_tablet_manager.cc:1398] T 9580f3fdf7084cc0923b1b16bfedd450 P 2db304b538e14857b7b1cc272d0db94b: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 14:26:27.842957 18559 catalog_manager.cc:5484] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2db304b538e14857b7b1cc272d0db94b (127.5.107.65). New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:26:27.875854 19707 raft_consensus.cc:482] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:27.876049 19707 raft_consensus.cc:504] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:27.877782 19707 leader_election.cc:288] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409), 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:26:27.884752 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9580f3fdf7084cc0923b1b16bfedd450" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b" is_pre_election: true
I20240502 14:26:27.884810 18965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9580f3fdf7084cc0923b1b16bfedd450" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06" is_pre_election: true
I20240502 14:26:27.885164 18701 raft_consensus.cc:2454] T 9580f3fdf7084cc0923b1b16bfedd450 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 0.
I20240502 14:26:27.885291 18965 raft_consensus.cc:2454] T 9580f3fdf7084cc0923b1b16bfedd450 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 0.
I20240502 14:26:27.885787 18766 leader_election.cc:302] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b, fec04c50dc5b49878ffa3c3873df32e4; no voters:
I20240502 14:26:27.886179 19707 raft_consensus.cc:2781] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:27.886303 19707 raft_consensus.cc:482] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:27.886425 19707 raft_consensus.cc:3037] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:27.888371 19707 raft_consensus.cc:504] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:27.888980 19707 leader_election.cc:288] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 election: Requested vote from peers c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409), 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:26:27.889459 18965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9580f3fdf7084cc0923b1b16bfedd450" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06"
I20240502 14:26:27.889529 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9580f3fdf7084cc0923b1b16bfedd450" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b"
I20240502 14:26:27.889667 18965 raft_consensus.cc:3037] T 9580f3fdf7084cc0923b1b16bfedd450 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:27.889732 18701 raft_consensus.cc:3037] T 9580f3fdf7084cc0923b1b16bfedd450 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:27.891616 18701 raft_consensus.cc:2454] T 9580f3fdf7084cc0923b1b16bfedd450 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 1.
I20240502 14:26:27.891613 18965 raft_consensus.cc:2454] T 9580f3fdf7084cc0923b1b16bfedd450 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 1.
I20240502 14:26:27.892072 18769 leader_election.cc:302] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c902805060df4df9b5599d0e94b2dc06, fec04c50dc5b49878ffa3c3873df32e4; no voters:
I20240502 14:26:27.892427 19707 raft_consensus.cc:2781] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:26:27.892899 19707 raft_consensus.cc:686] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Becoming Leader. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Running, Role: LEADER
I20240502 14:26:27.893440 19707 consensus_queue.cc:237] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:27.897673 18559 catalog_manager.cc:5484] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66). New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } }
W20240502 14:26:27.904753 18880 tablet.cc:2369] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:26:27.974447 18748 tablet.cc:2369] T 9580f3fdf7084cc0923b1b16bfedd450 P 2db304b538e14857b7b1cc272d0db94b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:26:27.985052 19706 raft_consensus.cc:482] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:27.985291 19706 raft_consensus.cc:504] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:27.986974 19706 leader_election.cc:288] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463), 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:26:27.993929 19097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2a6426ae6c438f942d39530da6e3e3" candidate_uuid: "c902805060df4df9b5599d0e94b2dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b2e79dc424a6f80141180f072f2d8" is_pre_election: true
I20240502 14:26:27.993916 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2a6426ae6c438f942d39530da6e3e3" candidate_uuid: "c902805060df4df9b5599d0e94b2dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b" is_pre_election: true
I20240502 14:26:27.994181 19097 raft_consensus.cc:2454] T cf2a6426ae6c438f942d39530da6e3e3 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c902805060df4df9b5599d0e94b2dc06 in term 0.
I20240502 14:26:27.994208 18701 raft_consensus.cc:2454] T cf2a6426ae6c438f942d39530da6e3e3 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c902805060df4df9b5599d0e94b2dc06 in term 0.
I20240502 14:26:27.994664 18898 leader_election.cc:302] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b, c902805060df4df9b5599d0e94b2dc06; no voters:
I20240502 14:26:27.995097 19706 raft_consensus.cc:2781] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:27.995224 19706 raft_consensus.cc:482] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:27.995317 19706 raft_consensus.cc:3037] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:27.997421 19706 raft_consensus.cc:504] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:27.998126 19706 leader_election.cc:288] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 1 election: Requested vote from peers 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463), 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:26:27.998577 19097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2a6426ae6c438f942d39530da6e3e3" candidate_uuid: "c902805060df4df9b5599d0e94b2dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b2e79dc424a6f80141180f072f2d8"
W20240502 14:26:27.998721 19143 tablet.cc:2369] T cf2a6426ae6c438f942d39530da6e3e3 P 007b2e79dc424a6f80141180f072f2d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:26:27.998700 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2a6426ae6c438f942d39530da6e3e3" candidate_uuid: "c902805060df4df9b5599d0e94b2dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b"
I20240502 14:26:27.998849 19097 raft_consensus.cc:3037] T cf2a6426ae6c438f942d39530da6e3e3 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:27.998965 18701 raft_consensus.cc:3037] T cf2a6426ae6c438f942d39530da6e3e3 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:28.000820 18701 raft_consensus.cc:2454] T cf2a6426ae6c438f942d39530da6e3e3 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c902805060df4df9b5599d0e94b2dc06 in term 1.
I20240502 14:26:28.000880 19097 raft_consensus.cc:2454] T cf2a6426ae6c438f942d39530da6e3e3 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c902805060df4df9b5599d0e94b2dc06 in term 1.
I20240502 14:26:28.001243 18898 leader_election.cc:302] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b, c902805060df4df9b5599d0e94b2dc06; no voters:
I20240502 14:26:28.001577 19706 raft_consensus.cc:2781] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:26:28.002009 19706 raft_consensus.cc:686] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Becoming Leader. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Running, Role: LEADER
I20240502 14:26:28.002597 19706 consensus_queue.cc:237] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:28.006558 18559 catalog_manager.cc:5484] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 reported cstate change: term changed from 0 to 1, leader changed from <none> to c902805060df4df9b5599d0e94b2dc06 (127.5.107.67). New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:26:28.287406 19707 consensus_queue.cc:1035] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:28.294152 19707 consensus_queue.cc:1035] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:28.349041 19716 consensus_queue.cc:1035] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:28.368758 19735 consensus_queue.cc:1035] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:26:28.417353 19727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:26:28.417688 19727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:26:28.425184 19727 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:26:28.445215 19706 consensus_queue.cc:1035] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:28.450099 19706 consensus_queue.cc:1035] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 232.676 ms
time per row: 0.116338 ms
I20240502 14:26:28.805761 18552 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48228:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:26:28.814957 18812 tablet_service.cc:1467] Processing CreateTablet for tablet 8937c809610542d88fdde3daf002c890 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3670a8bef9694c029d2128006cc913d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:28.815191 18945 tablet_service.cc:1467] Processing CreateTablet for tablet 8937c809610542d88fdde3daf002c890 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3670a8bef9694c029d2128006cc913d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:28.815500 18812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8937c809610542d88fdde3daf002c890. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:28.815632 18945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8937c809610542d88fdde3daf002c890. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:28.816406 19077 tablet_service.cc:1467] Processing CreateTablet for tablet 8937c809610542d88fdde3daf002c890 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3670a8bef9694c029d2128006cc913d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:28.816857 19077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8937c809610542d88fdde3daf002c890. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:28.817324 18811 tablet_service.cc:1467] Processing CreateTablet for tablet 0ffb6e582ef04fb38dd273c13febac58 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3670a8bef9694c029d2128006cc913d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:28.817781 18811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ffb6e582ef04fb38dd273c13febac58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:28.821573 18813 tablet_service.cc:1467] Processing CreateTablet for tablet 6acde0e5509340fbb52bca18645a77b7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3670a8bef9694c029d2128006cc913d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:28.821962 18813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6acde0e5509340fbb52bca18645a77b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:28.821898 18681 tablet_service.cc:1467] Processing CreateTablet for tablet 0ffb6e582ef04fb38dd273c13febac58 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3670a8bef9694c029d2128006cc913d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:28.822336 18681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ffb6e582ef04fb38dd273c13febac58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:28.822542 18943 tablet_service.cc:1467] Processing CreateTablet for tablet 0ffb6e582ef04fb38dd273c13febac58 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3670a8bef9694c029d2128006cc913d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:28.822969 18943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ffb6e582ef04fb38dd273c13febac58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:28.826195 19076 tablet_service.cc:1467] Processing CreateTablet for tablet 6acde0e5509340fbb52bca18645a77b7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3670a8bef9694c029d2128006cc913d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:28.826615 19076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6acde0e5509340fbb52bca18645a77b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:28.828871 18944 tablet_service.cc:1467] Processing CreateTablet for tablet 6acde0e5509340fbb52bca18645a77b7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3670a8bef9694c029d2128006cc913d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:28.829319 18944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6acde0e5509340fbb52bca18645a77b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:28.833595 19796 tablet_bootstrap.cc:492] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b: Bootstrap starting.
I20240502 14:26:28.835570 19798 tablet_bootstrap.cc:492] T 8937c809610542d88fdde3daf002c890 P fec04c50dc5b49878ffa3c3873df32e4: Bootstrap starting.
I20240502 14:26:28.835606 19796 tablet_bootstrap.cc:654] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:28.837862 19798 tablet_bootstrap.cc:654] T 8937c809610542d88fdde3daf002c890 P fec04c50dc5b49878ffa3c3873df32e4: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:28.839500 19795 tablet_bootstrap.cc:492] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8: Bootstrap starting.
I20240502 14:26:28.841032 19796 tablet_bootstrap.cc:492] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b: No bootstrap required, opened a new log
I20240502 14:26:28.841284 19796 ts_tablet_manager.cc:1367] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20240502 14:26:28.841698 19795 tablet_bootstrap.cc:654] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:28.842465 19796 raft_consensus.cc:348] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:28.842774 19796 raft_consensus.cc:374] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:28.842867 19796 raft_consensus.cc:729] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Initialized, Role: FOLLOWER
I20240502 14:26:28.843021 19797 tablet_bootstrap.cc:492] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06: Bootstrap starting.
I20240502 14:26:28.843201 19796 consensus_queue.cc:260] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:28.843919 19796 ts_tablet_manager.cc:1398] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:26:28.846974 19798 tablet_bootstrap.cc:492] T 8937c809610542d88fdde3daf002c890 P fec04c50dc5b49878ffa3c3873df32e4: No bootstrap required, opened a new log
I20240502 14:26:28.847193 19798 ts_tablet_manager.cc:1367] T 8937c809610542d88fdde3daf002c890 P fec04c50dc5b49878ffa3c3873df32e4: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20240502 14:26:28.848152 19798 raft_consensus.cc:348] T 8937c809610542d88fdde3daf002c890 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:28.848327 19798 raft_consensus.cc:374] T 8937c809610542d88fdde3daf002c890 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:28.848426 19798 raft_consensus.cc:729] T 8937c809610542d88fdde3daf002c890 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Initialized, Role: FOLLOWER
I20240502 14:26:28.848659 19798 consensus_queue.cc:260] T 8937c809610542d88fdde3daf002c890 P fec04c50dc5b49878ffa3c3873df32e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:28.849105 19798 ts_tablet_manager.cc:1398] T 8937c809610542d88fdde3daf002c890 P fec04c50dc5b49878ffa3c3873df32e4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:26:28.849444 19798 tablet_bootstrap.cc:492] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4: Bootstrap starting.
I20240502 14:26:28.849798 19797 tablet_bootstrap.cc:654] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:28.850526 19795 tablet_bootstrap.cc:492] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8: No bootstrap required, opened a new log
I20240502 14:26:28.850718 19795 ts_tablet_manager.cc:1367] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240502 14:26:28.851055 19798 tablet_bootstrap.cc:654] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:28.851873 19795 raft_consensus.cc:348] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:28.852154 19795 raft_consensus.cc:374] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:28.852250 19795 raft_consensus.cc:729] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Initialized, Role: FOLLOWER
I20240502 14:26:28.852888 19795 consensus_queue.cc:260] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:28.855141 19795 ts_tablet_manager.cc:1398] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240502 14:26:28.855501 19795 tablet_bootstrap.cc:492] T 6acde0e5509340fbb52bca18645a77b7 P 007b2e79dc424a6f80141180f072f2d8: Bootstrap starting.
I20240502 14:26:28.859227 19797 tablet_bootstrap.cc:492] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06: No bootstrap required, opened a new log
I20240502 14:26:28.859311 19795 tablet_bootstrap.cc:654] T 6acde0e5509340fbb52bca18645a77b7 P 007b2e79dc424a6f80141180f072f2d8: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:28.859423 19797 ts_tablet_manager.cc:1367] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
I20240502 14:26:28.860433 19797 raft_consensus.cc:348] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:28.860734 19797 raft_consensus.cc:374] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:28.860766 19798 tablet_bootstrap.cc:492] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4: No bootstrap required, opened a new log
I20240502 14:26:28.860847 19797 raft_consensus.cc:729] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Initialized, Role: FOLLOWER
I20240502 14:26:28.860951 19798 ts_tablet_manager.cc:1367] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.001s
I20240502 14:26:28.861151 19797 consensus_queue.cc:260] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:28.861886 19797 ts_tablet_manager.cc:1398] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:26:28.861827 19798 raft_consensus.cc:348] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:28.862010 19798 raft_consensus.cc:374] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:28.862082 19798 raft_consensus.cc:729] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Initialized, Role: FOLLOWER
I20240502 14:26:28.862188 19797 tablet_bootstrap.cc:492] T 6acde0e5509340fbb52bca18645a77b7 P c902805060df4df9b5599d0e94b2dc06: Bootstrap starting.
I20240502 14:26:28.862323 19798 consensus_queue.cc:260] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:28.862799 19795 tablet_bootstrap.cc:492] T 6acde0e5509340fbb52bca18645a77b7 P 007b2e79dc424a6f80141180f072f2d8: No bootstrap required, opened a new log
I20240502 14:26:28.862960 19795 ts_tablet_manager.cc:1367] T 6acde0e5509340fbb52bca18645a77b7 P 007b2e79dc424a6f80141180f072f2d8: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20240502 14:26:28.862893 19798 ts_tablet_manager.cc:1398] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:26:28.863592 19798 tablet_bootstrap.cc:492] T 0ffb6e582ef04fb38dd273c13febac58 P fec04c50dc5b49878ffa3c3873df32e4: Bootstrap starting.
I20240502 14:26:28.863816 19795 raft_consensus.cc:348] T 6acde0e5509340fbb52bca18645a77b7 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:28.864048 19795 raft_consensus.cc:374] T 6acde0e5509340fbb52bca18645a77b7 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:28.864131 19795 raft_consensus.cc:729] T 6acde0e5509340fbb52bca18645a77b7 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Initialized, Role: FOLLOWER
I20240502 14:26:28.864413 19795 consensus_queue.cc:260] T 6acde0e5509340fbb52bca18645a77b7 P 007b2e79dc424a6f80141180f072f2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:28.865005 19798 tablet_bootstrap.cc:654] T 0ffb6e582ef04fb38dd273c13febac58 P fec04c50dc5b49878ffa3c3873df32e4: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:28.868903 19795 ts_tablet_manager.cc:1398] T 6acde0e5509340fbb52bca18645a77b7 P 007b2e79dc424a6f80141180f072f2d8: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 14:26:28.871474 19797 tablet_bootstrap.cc:654] T 6acde0e5509340fbb52bca18645a77b7 P c902805060df4df9b5599d0e94b2dc06: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:28.873549 19797 tablet_bootstrap.cc:492] T 6acde0e5509340fbb52bca18645a77b7 P c902805060df4df9b5599d0e94b2dc06: No bootstrap required, opened a new log
I20240502 14:26:28.873725 19797 ts_tablet_manager.cc:1367] T 6acde0e5509340fbb52bca18645a77b7 P c902805060df4df9b5599d0e94b2dc06: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240502 14:26:28.873749 19798 tablet_bootstrap.cc:492] T 0ffb6e582ef04fb38dd273c13febac58 P fec04c50dc5b49878ffa3c3873df32e4: No bootstrap required, opened a new log
I20240502 14:26:28.873930 19798 ts_tablet_manager.cc:1367] T 0ffb6e582ef04fb38dd273c13febac58 P fec04c50dc5b49878ffa3c3873df32e4: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240502 14:26:28.874433 19797 raft_consensus.cc:348] T 6acde0e5509340fbb52bca18645a77b7 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:28.874631 19797 raft_consensus.cc:374] T 6acde0e5509340fbb52bca18645a77b7 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:28.874698 19797 raft_consensus.cc:729] T 6acde0e5509340fbb52bca18645a77b7 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Initialized, Role: FOLLOWER
I20240502 14:26:28.874590 19798 raft_consensus.cc:348] T 0ffb6e582ef04fb38dd273c13febac58 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:28.874778 19798 raft_consensus.cc:374] T 0ffb6e582ef04fb38dd273c13febac58 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:28.874861 19798 raft_consensus.cc:729] T 0ffb6e582ef04fb38dd273c13febac58 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Initialized, Role: FOLLOWER
I20240502 14:26:28.874949 19797 consensus_queue.cc:260] T 6acde0e5509340fbb52bca18645a77b7 P c902805060df4df9b5599d0e94b2dc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:28.875078 19798 consensus_queue.cc:260] T 0ffb6e582ef04fb38dd273c13febac58 P fec04c50dc5b49878ffa3c3873df32e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:28.875587 19798 ts_tablet_manager.cc:1398] T 0ffb6e582ef04fb38dd273c13febac58 P fec04c50dc5b49878ffa3c3873df32e4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:26:28.875921 19797 ts_tablet_manager.cc:1398] T 6acde0e5509340fbb52bca18645a77b7 P c902805060df4df9b5599d0e94b2dc06: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:26:28.876210 19797 tablet_bootstrap.cc:492] T 8937c809610542d88fdde3daf002c890 P c902805060df4df9b5599d0e94b2dc06: Bootstrap starting.
I20240502 14:26:28.877472 19797 tablet_bootstrap.cc:654] T 8937c809610542d88fdde3daf002c890 P c902805060df4df9b5599d0e94b2dc06: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:28.879747 19797 tablet_bootstrap.cc:492] T 8937c809610542d88fdde3daf002c890 P c902805060df4df9b5599d0e94b2dc06: No bootstrap required, opened a new log
I20240502 14:26:28.879894 19797 ts_tablet_manager.cc:1367] T 8937c809610542d88fdde3daf002c890 P c902805060df4df9b5599d0e94b2dc06: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:26:28.880533 19797 raft_consensus.cc:348] T 8937c809610542d88fdde3daf002c890 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:28.880753 19797 raft_consensus.cc:374] T 8937c809610542d88fdde3daf002c890 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:28.880856 19797 raft_consensus.cc:729] T 8937c809610542d88fdde3daf002c890 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Initialized, Role: FOLLOWER
I20240502 14:26:28.881073 19797 consensus_queue.cc:260] T 8937c809610542d88fdde3daf002c890 P c902805060df4df9b5599d0e94b2dc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:28.881623 19797 ts_tablet_manager.cc:1398] T 8937c809610542d88fdde3daf002c890 P c902805060df4df9b5599d0e94b2dc06: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:26:28.934679 19735 raft_consensus.cc:482] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:28.934927 19735 raft_consensus.cc:504] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:28.935748 19735 leader_election.cc:288] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409), fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
I20240502 14:26:28.936429 18833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ffb6e582ef04fb38dd273c13febac58" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec04c50dc5b49878ffa3c3873df32e4" is_pre_election: true
I20240502 14:26:28.936684 18833 raft_consensus.cc:2454] T 0ffb6e582ef04fb38dd273c13febac58 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2db304b538e14857b7b1cc272d0db94b in term 0.
I20240502 14:26:28.937170 18634 leader_election.cc:302] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b, fec04c50dc5b49878ffa3c3873df32e4; no voters:
I20240502 14:26:28.937517 19735 raft_consensus.cc:2781] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:28.937594 19735 raft_consensus.cc:482] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:28.937716 19735 raft_consensus.cc:3037] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:28.940052 19735 raft_consensus.cc:504] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:28.940871 19735 leader_election.cc:288] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 election: Requested vote from peers c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409), fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
I20240502 14:26:28.941432 18833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ffb6e582ef04fb38dd273c13febac58" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec04c50dc5b49878ffa3c3873df32e4"
I20240502 14:26:28.941712 18833 raft_consensus.cc:3037] T 0ffb6e582ef04fb38dd273c13febac58 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:28.943336 18965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ffb6e582ef04fb38dd273c13febac58" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06" is_pre_election: true
I20240502 14:26:28.943388 18964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ffb6e582ef04fb38dd273c13febac58" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06"
I20240502 14:26:28.943648 18965 raft_consensus.cc:2454] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2db304b538e14857b7b1cc272d0db94b in term 0.
I20240502 14:26:28.944197 18833 raft_consensus.cc:2454] T 0ffb6e582ef04fb38dd273c13febac58 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2db304b538e14857b7b1cc272d0db94b in term 1.
I20240502 14:26:28.944664 18634 leader_election.cc:302] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b, fec04c50dc5b49878ffa3c3873df32e4; no voters: c902805060df4df9b5599d0e94b2dc06
I20240502 14:26:28.944957 19735 raft_consensus.cc:2781] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:26:28.945118 19735 raft_consensus.cc:686] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Becoming Leader. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Running, Role: LEADER
I20240502 14:26:28.945425 19735 consensus_queue.cc:237] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:28.948133 18552 catalog_manager.cc:5484] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2db304b538e14857b7b1cc272d0db94b (127.5.107.65). New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } }
I20240502 14:26:29.030750 19733 raft_consensus.cc:482] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:29.031008 19733 raft_consensus.cc:504] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:29.032753 19733 leader_election.cc:288] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009), c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:26:29.041728 18833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8937c809610542d88fdde3daf002c890" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec04c50dc5b49878ffa3c3873df32e4" is_pre_election: true
I20240502 14:26:29.041769 18964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8937c809610542d88fdde3daf002c890" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06" is_pre_election: true
I20240502 14:26:29.041985 18833 raft_consensus.cc:2454] T 8937c809610542d88fdde3daf002c890 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 0.
I20240502 14:26:29.042029 18964 raft_consensus.cc:2454] T 8937c809610542d88fdde3daf002c890 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 0.
I20240502 14:26:29.042593 19029 leader_election.cc:302] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 007b2e79dc424a6f80141180f072f2d8, fec04c50dc5b49878ffa3c3873df32e4; no voters:
I20240502 14:26:29.042806 19787 raft_consensus.cc:482] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:29.042974 19787 raft_consensus.cc:504] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:29.043341 19733 raft_consensus.cc:2781] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:29.043455 19733 raft_consensus.cc:482] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:29.043560 19733 raft_consensus.cc:3037] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:29.043864 19787 leader_election.cc:288] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463), c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:26:29.044982 18964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6acde0e5509340fbb52bca18645a77b7" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06" is_pre_election: true
I20240502 14:26:29.045222 18964 raft_consensus.cc:2454] T 6acde0e5509340fbb52bca18645a77b7 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 0.
I20240502 14:26:29.045629 18769 leader_election.cc:302] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c902805060df4df9b5599d0e94b2dc06, fec04c50dc5b49878ffa3c3873df32e4; no voters:
I20240502 14:26:29.045897 19787 raft_consensus.cc:2781] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:29.046010 19787 raft_consensus.cc:482] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:29.046108 19787 raft_consensus.cc:3037] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:29.046557 19733 raft_consensus.cc:504] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:29.047783 18964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8937c809610542d88fdde3daf002c890" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06"
I20240502 14:26:29.047808 18833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8937c809610542d88fdde3daf002c890" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec04c50dc5b49878ffa3c3873df32e4"
I20240502 14:26:29.048002 18833 raft_consensus.cc:3037] T 8937c809610542d88fdde3daf002c890 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:29.048014 18964 raft_consensus.cc:3037] T 8937c809610542d88fdde3daf002c890 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:29.048799 19787 raft_consensus.cc:504] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:29.049610 19787 leader_election.cc:288] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 election: Requested vote from peers 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463), c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:26:29.050087 18965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6acde0e5509340fbb52bca18645a77b7" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06"
I20240502 14:26:29.050311 18965 raft_consensus.cc:3037] T 6acde0e5509340fbb52bca18645a77b7 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:29.050307 19733 leader_election.cc:288] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 election: Requested vote from peers fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009), c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:26:29.051184 18833 raft_consensus.cc:2454] T 8937c809610542d88fdde3daf002c890 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 1.
I20240502 14:26:29.051419 18964 raft_consensus.cc:2454] T 8937c809610542d88fdde3daf002c890 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 1.
I20240502 14:26:29.051690 19029 leader_election.cc:302] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 007b2e79dc424a6f80141180f072f2d8, fec04c50dc5b49878ffa3c3873df32e4; no voters:
I20240502 14:26:29.052250 19733 raft_consensus.cc:2781] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:26:29.052815 19733 raft_consensus.cc:686] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Becoming Leader. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Running, Role: LEADER
I20240502 14:26:29.053131 18965 raft_consensus.cc:2454] T 6acde0e5509340fbb52bca18645a77b7 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 1.
I20240502 14:26:29.053681 18769 leader_election.cc:302] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c902805060df4df9b5599d0e94b2dc06, fec04c50dc5b49878ffa3c3873df32e4; no voters:
I20240502 14:26:29.053731 19733 consensus_queue.cc:237] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:29.054011 19787 raft_consensus.cc:2781] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:26:29.054188 19787 raft_consensus.cc:686] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Becoming Leader. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Running, Role: LEADER
I20240502 14:26:29.054510 19787 consensus_queue.cc:237] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:29.054606 19096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6acde0e5509340fbb52bca18645a77b7" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b2e79dc424a6f80141180f072f2d8" is_pre_election: true
I20240502 14:26:29.054832 19096 raft_consensus.cc:2454] T 6acde0e5509340fbb52bca18645a77b7 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 0.
I20240502 14:26:29.055061 19097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6acde0e5509340fbb52bca18645a77b7" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b2e79dc424a6f80141180f072f2d8"
I20240502 14:26:29.055281 19097 raft_consensus.cc:3037] T 6acde0e5509340fbb52bca18645a77b7 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:29.057948 18552 catalog_manager.cc:5484] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68). New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:26:29.058560 19097 raft_consensus.cc:2454] T 6acde0e5509340fbb52bca18645a77b7 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 1.
I20240502 14:26:29.061514 18560 catalog_manager.cc:5484] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66). New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
W20240502 14:26:29.430927 19817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:26:29.431319 19817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 14:26:29.435078 19787 consensus_queue.cc:1035] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:29.436040 18965 raft_consensus.cc:3037] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:29.439915 19735 consensus_queue.cc:1035] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:26:29.442270 19817 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:26:29.447851 19787 consensus_queue.cc:1035] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:29.459203 19735 consensus_queue.cc:1035] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240502 14:26:29.609340 19733 consensus_queue.cc:1035] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:26:29.680130 19029 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.66:33009, user_credentials={real_user=slave}} blocked reactor thread for 71018.4us
I20240502 14:26:29.717345 19733 consensus_queue.cc:1035] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 265.593 ms
time per row: 0.132796 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:26:30.773409 18559 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48228:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:26:30.782027 18681 tablet_service.cc:1467] Processing CreateTablet for tablet 0d8667d91fd64809b635362c4a8fa33f (DEFAULT_TABLE table=rebalance_test_table_2 [id=0e84c82d60e64dbd9149c5f9fb7ff3c8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:30.782363 18812 tablet_service.cc:1467] Processing CreateTablet for tablet 0d8667d91fd64809b635362c4a8fa33f (DEFAULT_TABLE table=rebalance_test_table_2 [id=0e84c82d60e64dbd9149c5f9fb7ff3c8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:30.782496 18681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d8667d91fd64809b635362c4a8fa33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:30.782819 18812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d8667d91fd64809b635362c4a8fa33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:30.783824 19076 tablet_service.cc:1467] Processing CreateTablet for tablet 96662a3d9e1b4f03b05464ddd06b93d6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0e84c82d60e64dbd9149c5f9fb7ff3c8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:30.784271 19076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96662a3d9e1b4f03b05464ddd06b93d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:30.784200 18679 tablet_service.cc:1467] Processing CreateTablet for tablet 522014370f0a4cc3bd31aa00d6a05c5e (DEFAULT_TABLE table=rebalance_test_table_2 [id=0e84c82d60e64dbd9149c5f9fb7ff3c8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:30.784546 18679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 522014370f0a4cc3bd31aa00d6a05c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:30.789597 19075 tablet_service.cc:1467] Processing CreateTablet for tablet 522014370f0a4cc3bd31aa00d6a05c5e (DEFAULT_TABLE table=rebalance_test_table_2 [id=0e84c82d60e64dbd9149c5f9fb7ff3c8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:30.789991 19075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 522014370f0a4cc3bd31aa00d6a05c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:30.793674 18680 tablet_service.cc:1467] Processing CreateTablet for tablet 96662a3d9e1b4f03b05464ddd06b93d6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0e84c82d60e64dbd9149c5f9fb7ff3c8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:30.794170 18680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96662a3d9e1b4f03b05464ddd06b93d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:30.795652 19888 tablet_bootstrap.cc:492] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4: Bootstrap starting.
I20240502 14:26:30.796816 19888 tablet_bootstrap.cc:654] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:30.797745 18943 tablet_service.cc:1467] Processing CreateTablet for tablet 96662a3d9e1b4f03b05464ddd06b93d6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0e84c82d60e64dbd9149c5f9fb7ff3c8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:30.798252 18943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96662a3d9e1b4f03b05464ddd06b93d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:30.799209 19889 tablet_bootstrap.cc:492] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8: Bootstrap starting.
I20240502 14:26:30.800221 19888 tablet_bootstrap.cc:492] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4: No bootstrap required, opened a new log
I20240502 14:26:30.800354 19888 ts_tablet_manager.cc:1367] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240502 14:26:30.800693 19889 tablet_bootstrap.cc:654] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:30.801055 19888 raft_consensus.cc:348] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:30.801242 19888 raft_consensus.cc:374] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:30.801330 19888 raft_consensus.cc:729] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Initialized, Role: FOLLOWER
I20240502 14:26:30.801563 19077 tablet_service.cc:1467] Processing CreateTablet for tablet 0d8667d91fd64809b635362c4a8fa33f (DEFAULT_TABLE table=rebalance_test_table_2 [id=0e84c82d60e64dbd9149c5f9fb7ff3c8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:30.802017 19077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d8667d91fd64809b635362c4a8fa33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:30.802328 19887 tablet_bootstrap.cc:492] T 0d8667d91fd64809b635362c4a8fa33f P 2db304b538e14857b7b1cc272d0db94b: Bootstrap starting.
I20240502 14:26:30.803483 19889 tablet_bootstrap.cc:492] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8: No bootstrap required, opened a new log
I20240502 14:26:30.803633 19889 ts_tablet_manager.cc:1367] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 14:26:30.803711 19887 tablet_bootstrap.cc:654] T 0d8667d91fd64809b635362c4a8fa33f P 2db304b538e14857b7b1cc272d0db94b: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:30.804438 19889 raft_consensus.cc:348] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:30.804627 19889 raft_consensus.cc:374] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:30.804711 19889 raft_consensus.cc:729] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Initialized, Role: FOLLOWER
I20240502 14:26:30.804949 19889 consensus_queue.cc:260] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:30.805084 19888 consensus_queue.cc:260] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:30.805531 19889 ts_tablet_manager.cc:1398] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:26:30.805918 19888 ts_tablet_manager.cc:1398] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:26:30.806349 18812 tablet_service.cc:1467] Processing CreateTablet for tablet 522014370f0a4cc3bd31aa00d6a05c5e (DEFAULT_TABLE table=rebalance_test_table_2 [id=0e84c82d60e64dbd9149c5f9fb7ff3c8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:30.806774 18812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 522014370f0a4cc3bd31aa00d6a05c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:30.807520 19887 tablet_bootstrap.cc:492] T 0d8667d91fd64809b635362c4a8fa33f P 2db304b538e14857b7b1cc272d0db94b: No bootstrap required, opened a new log
I20240502 14:26:30.807700 19887 ts_tablet_manager.cc:1367] T 0d8667d91fd64809b635362c4a8fa33f P 2db304b538e14857b7b1cc272d0db94b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:26:30.808488 19887 raft_consensus.cc:348] T 0d8667d91fd64809b635362c4a8fa33f P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:30.808714 19887 raft_consensus.cc:374] T 0d8667d91fd64809b635362c4a8fa33f P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:30.808835 19887 raft_consensus.cc:729] T 0d8667d91fd64809b635362c4a8fa33f P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Initialized, Role: FOLLOWER
I20240502 14:26:30.809073 19887 consensus_queue.cc:260] T 0d8667d91fd64809b635362c4a8fa33f P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:30.809963 19887 ts_tablet_manager.cc:1398] T 0d8667d91fd64809b635362c4a8fa33f P 2db304b538e14857b7b1cc272d0db94b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:26:30.812856 19888 tablet_bootstrap.cc:492] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4: Bootstrap starting.
I20240502 14:26:30.814591 19888 tablet_bootstrap.cc:654] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:30.819324 19889 tablet_bootstrap.cc:492] T 0d8667d91fd64809b635362c4a8fa33f P 007b2e79dc424a6f80141180f072f2d8: Bootstrap starting.
I20240502 14:26:30.819324 19887 tablet_bootstrap.cc:492] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b: Bootstrap starting.
I20240502 14:26:30.820922 19887 tablet_bootstrap.cc:654] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:30.820959 19889 tablet_bootstrap.cc:654] T 0d8667d91fd64809b635362c4a8fa33f P 007b2e79dc424a6f80141180f072f2d8: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:30.821640 19894 tablet_bootstrap.cc:492] T 96662a3d9e1b4f03b05464ddd06b93d6 P c902805060df4df9b5599d0e94b2dc06: Bootstrap starting.
I20240502 14:26:30.821943 19888 tablet_bootstrap.cc:492] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4: No bootstrap required, opened a new log
I20240502 14:26:30.822113 19888 ts_tablet_manager.cc:1367] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240502 14:26:30.822896 19888 raft_consensus.cc:348] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:30.823135 19888 raft_consensus.cc:374] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:30.823206 19888 raft_consensus.cc:729] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Initialized, Role: FOLLOWER
I20240502 14:26:30.823457 19888 consensus_queue.cc:260] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:30.823745 19894 tablet_bootstrap.cc:654] T 96662a3d9e1b4f03b05464ddd06b93d6 P c902805060df4df9b5599d0e94b2dc06: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:30.825326 19888 ts_tablet_manager.cc:1398] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:26:30.827891 19889 tablet_bootstrap.cc:492] T 0d8667d91fd64809b635362c4a8fa33f P 007b2e79dc424a6f80141180f072f2d8: No bootstrap required, opened a new log
I20240502 14:26:30.828083 19889 ts_tablet_manager.cc:1367] T 0d8667d91fd64809b635362c4a8fa33f P 007b2e79dc424a6f80141180f072f2d8: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20240502 14:26:30.830430 19889 raft_consensus.cc:348] T 0d8667d91fd64809b635362c4a8fa33f P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:30.830724 19889 raft_consensus.cc:374] T 0d8667d91fd64809b635362c4a8fa33f P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:30.830816 19889 raft_consensus.cc:729] T 0d8667d91fd64809b635362c4a8fa33f P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Initialized, Role: FOLLOWER
I20240502 14:26:30.831110 19889 consensus_queue.cc:260] T 0d8667d91fd64809b635362c4a8fa33f P 007b2e79dc424a6f80141180f072f2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:30.831781 19889 ts_tablet_manager.cc:1398] T 0d8667d91fd64809b635362c4a8fa33f P 007b2e79dc424a6f80141180f072f2d8: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 14:26:30.832147 19889 tablet_bootstrap.cc:492] T 522014370f0a4cc3bd31aa00d6a05c5e P 007b2e79dc424a6f80141180f072f2d8: Bootstrap starting.
I20240502 14:26:30.834300 19889 tablet_bootstrap.cc:654] T 522014370f0a4cc3bd31aa00d6a05c5e P 007b2e79dc424a6f80141180f072f2d8: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:30.837064 19887 tablet_bootstrap.cc:492] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b: No bootstrap required, opened a new log
I20240502 14:26:30.837277 19887 ts_tablet_manager.cc:1367] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.014s
I20240502 14:26:30.837666 19894 tablet_bootstrap.cc:492] T 96662a3d9e1b4f03b05464ddd06b93d6 P c902805060df4df9b5599d0e94b2dc06: No bootstrap required, opened a new log
I20240502 14:26:30.837832 19889 tablet_bootstrap.cc:492] T 522014370f0a4cc3bd31aa00d6a05c5e P 007b2e79dc424a6f80141180f072f2d8: No bootstrap required, opened a new log
I20240502 14:26:30.837824 19894 ts_tablet_manager.cc:1367] T 96662a3d9e1b4f03b05464ddd06b93d6 P c902805060df4df9b5599d0e94b2dc06: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.010s
I20240502 14:26:30.837955 19889 ts_tablet_manager.cc:1367] T 522014370f0a4cc3bd31aa00d6a05c5e P 007b2e79dc424a6f80141180f072f2d8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240502 14:26:30.837920 19887 raft_consensus.cc:348] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:30.838135 19887 raft_consensus.cc:374] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:30.838217 19887 raft_consensus.cc:729] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Initialized, Role: FOLLOWER
I20240502 14:26:30.838449 19887 consensus_queue.cc:260] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:30.838596 19889 raft_consensus.cc:348] T 522014370f0a4cc3bd31aa00d6a05c5e P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:30.838570 19894 raft_consensus.cc:348] T 96662a3d9e1b4f03b05464ddd06b93d6 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:30.838770 19889 raft_consensus.cc:374] T 522014370f0a4cc3bd31aa00d6a05c5e P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:30.838781 19894 raft_consensus.cc:374] T 96662a3d9e1b4f03b05464ddd06b93d6 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:30.838856 19889 raft_consensus.cc:729] T 522014370f0a4cc3bd31aa00d6a05c5e P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Initialized, Role: FOLLOWER
I20240502 14:26:30.838862 19894 raft_consensus.cc:729] T 96662a3d9e1b4f03b05464ddd06b93d6 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Initialized, Role: FOLLOWER
I20240502 14:26:30.839000 19887 ts_tablet_manager.cc:1398] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:26:30.839061 19894 consensus_queue.cc:260] T 96662a3d9e1b4f03b05464ddd06b93d6 P c902805060df4df9b5599d0e94b2dc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:30.839143 19889 consensus_queue.cc:260] T 522014370f0a4cc3bd31aa00d6a05c5e P 007b2e79dc424a6f80141180f072f2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:30.839414 19887 tablet_bootstrap.cc:492] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b: Bootstrap starting.
I20240502 14:26:30.839574 19894 ts_tablet_manager.cc:1398] T 96662a3d9e1b4f03b05464ddd06b93d6 P c902805060df4df9b5599d0e94b2dc06: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:26:30.840358 19889 ts_tablet_manager.cc:1398] T 522014370f0a4cc3bd31aa00d6a05c5e P 007b2e79dc424a6f80141180f072f2d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:26:30.840970 19887 tablet_bootstrap.cc:654] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:30.846726 19887 tablet_bootstrap.cc:492] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b: No bootstrap required, opened a new log
I20240502 14:26:30.846896 19887 ts_tablet_manager.cc:1367] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20240502 14:26:30.847451 19887 raft_consensus.cc:348] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:30.847594 19887 raft_consensus.cc:374] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:30.847684 19887 raft_consensus.cc:729] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Initialized, Role: FOLLOWER
I20240502 14:26:30.847864 19887 consensus_queue.cc:260] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:30.848323 19887 ts_tablet_manager.cc:1398] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:26:30.897797 19882 raft_consensus.cc:482] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:30.898025 19882 raft_consensus.cc:504] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:30.898772 19882 leader_election.cc:288] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749), 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:26:30.899340 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8667d91fd64809b635362c4a8fa33f" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b" is_pre_election: true
I20240502 14:26:30.899408 19096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8667d91fd64809b635362c4a8fa33f" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b2e79dc424a6f80141180f072f2d8" is_pre_election: true
I20240502 14:26:30.899621 19096 raft_consensus.cc:2454] T 0d8667d91fd64809b635362c4a8fa33f P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 0.
I20240502 14:26:30.899613 18701 raft_consensus.cc:2454] T 0d8667d91fd64809b635362c4a8fa33f P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 0.
I20240502 14:26:30.900102 18766 leader_election.cc:302] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b, fec04c50dc5b49878ffa3c3873df32e4; no voters:
I20240502 14:26:30.900422 19882 raft_consensus.cc:2781] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:30.900552 19882 raft_consensus.cc:482] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:30.900636 19882 raft_consensus.cc:3037] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:30.903590 19882 raft_consensus.cc:504] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:30.904246 19882 leader_election.cc:288] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 election: Requested vote from peers 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749), 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:26:30.904765 19096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8667d91fd64809b635362c4a8fa33f" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b2e79dc424a6f80141180f072f2d8"
I20240502 14:26:30.904794 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d8667d91fd64809b635362c4a8fa33f" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b"
I20240502 14:26:30.904963 19096 raft_consensus.cc:3037] T 0d8667d91fd64809b635362c4a8fa33f P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:30.904974 18701 raft_consensus.cc:3037] T 0d8667d91fd64809b635362c4a8fa33f P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:30.907023 18701 raft_consensus.cc:2454] T 0d8667d91fd64809b635362c4a8fa33f P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 1.
I20240502 14:26:30.907456 18766 leader_election.cc:302] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b, fec04c50dc5b49878ffa3c3873df32e4; no voters:
I20240502 14:26:30.907757 19882 raft_consensus.cc:2781] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:26:30.907750 19096 raft_consensus.cc:2454] T 0d8667d91fd64809b635362c4a8fa33f P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 1.
I20240502 14:26:30.908002 19882 raft_consensus.cc:686] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Becoming Leader. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Running, Role: LEADER
I20240502 14:26:30.908685 19882 consensus_queue.cc:237] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:30.911684 18559 catalog_manager.cc:5484] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66). New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:26:30.935580 19825 raft_consensus.cc:482] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:30.935807 19825 raft_consensus.cc:504] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:30.936486 19825 leader_election.cc:288] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409), 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:26:30.936959 18964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96662a3d9e1b4f03b05464ddd06b93d6" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06" is_pre_election: true
I20240502 14:26:30.936976 19096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96662a3d9e1b4f03b05464ddd06b93d6" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b2e79dc424a6f80141180f072f2d8" is_pre_election: true
I20240502 14:26:30.937175 18964 raft_consensus.cc:2454] T 96662a3d9e1b4f03b05464ddd06b93d6 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2db304b538e14857b7b1cc272d0db94b in term 0.
I20240502 14:26:30.937233 19096 raft_consensus.cc:2454] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2db304b538e14857b7b1cc272d0db94b in term 0.
I20240502 14:26:30.937664 18637 leader_election.cc:302] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b, c902805060df4df9b5599d0e94b2dc06; no voters:
I20240502 14:26:30.937922 19825 raft_consensus.cc:2781] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:30.938038 19825 raft_consensus.cc:482] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:30.938130 19825 raft_consensus.cc:3037] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:30.939951 19825 raft_consensus.cc:504] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:30.940490 19825 leader_election.cc:288] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 election: Requested vote from peers c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409), 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:26:30.940932 18964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96662a3d9e1b4f03b05464ddd06b93d6" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06"
I20240502 14:26:30.940982 19096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96662a3d9e1b4f03b05464ddd06b93d6" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b2e79dc424a6f80141180f072f2d8"
I20240502 14:26:30.941103 18964 raft_consensus.cc:3037] T 96662a3d9e1b4f03b05464ddd06b93d6 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:30.941226 19096 raft_consensus.cc:3037] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:30.943172 19096 raft_consensus.cc:2454] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2db304b538e14857b7b1cc272d0db94b in term 1.
I20240502 14:26:30.943233 18964 raft_consensus.cc:2454] T 96662a3d9e1b4f03b05464ddd06b93d6 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2db304b538e14857b7b1cc272d0db94b in term 1.
I20240502 14:26:30.943578 18637 leader_election.cc:302] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 007b2e79dc424a6f80141180f072f2d8, 2db304b538e14857b7b1cc272d0db94b; no voters:
I20240502 14:26:30.943857 19825 raft_consensus.cc:2781] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:26:30.944223 19825 raft_consensus.cc:686] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Becoming Leader. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Running, Role: LEADER
I20240502 14:26:30.944512 19825 consensus_queue.cc:237] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:30.947216 18559 catalog_manager.cc:5484] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2db304b538e14857b7b1cc272d0db94b (127.5.107.65). New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:26:30.980091 19882 raft_consensus.cc:482] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:30.980335 19882 raft_consensus.cc:504] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:30.981148 19882 leader_election.cc:288] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463), 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:26:30.981753 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "522014370f0a4cc3bd31aa00d6a05c5e" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b" is_pre_election: true
I20240502 14:26:30.981771 19096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "522014370f0a4cc3bd31aa00d6a05c5e" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b2e79dc424a6f80141180f072f2d8" is_pre_election: true
I20240502 14:26:30.981984 19096 raft_consensus.cc:2454] T 522014370f0a4cc3bd31aa00d6a05c5e P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 0.
I20240502 14:26:30.982012 18701 raft_consensus.cc:2454] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 0.
I20240502 14:26:30.982455 18769 leader_election.cc:302] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 007b2e79dc424a6f80141180f072f2d8, 2db304b538e14857b7b1cc272d0db94b, fec04c50dc5b49878ffa3c3873df32e4; no voters:
I20240502 14:26:30.982795 19882 raft_consensus.cc:2781] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:30.982887 19882 raft_consensus.cc:482] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:30.983000 19882 raft_consensus.cc:3037] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:30.984783 19882 raft_consensus.cc:504] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:30.985436 19882 leader_election.cc:288] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 election: Requested vote from peers 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463), 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:26:30.985916 19096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "522014370f0a4cc3bd31aa00d6a05c5e" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b2e79dc424a6f80141180f072f2d8"
I20240502 14:26:30.985939 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "522014370f0a4cc3bd31aa00d6a05c5e" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b"
I20240502 14:26:30.986105 19096 raft_consensus.cc:3037] T 522014370f0a4cc3bd31aa00d6a05c5e P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:30.986125 18701 raft_consensus.cc:3037] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:30.987963 18701 raft_consensus.cc:2454] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 1.
I20240502 14:26:30.987980 19096 raft_consensus.cc:2454] T 522014370f0a4cc3bd31aa00d6a05c5e P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 1.
I20240502 14:26:30.988368 18766 leader_election.cc:302] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b, fec04c50dc5b49878ffa3c3873df32e4; no voters:
I20240502 14:26:30.988664 19882 raft_consensus.cc:2781] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:26:30.988839 19882 raft_consensus.cc:686] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Becoming Leader. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Running, Role: LEADER
I20240502 14:26:30.989130 19882 consensus_queue.cc:237] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:30.992519 18559 catalog_manager.cc:5484] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66). New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:26:31.349856 19882 consensus_queue.cc:1035] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:31.354816 19882 consensus_queue.cc:1035] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:26:31.355412 19904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:26:31.355872 19904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:26:31.364662 19904 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:26:31.387504 19825 consensus_queue.cc:1035] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:31.393056 19825 consensus_queue.cc:1035] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240502 14:26:31.437708 19882 consensus_queue.cc:1035] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:31.445673 19882 consensus_queue.cc:1035] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 262.704 ms
time per row: 0.131352 ms
I20240502 14:26:31.780380 18552 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48228:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:26:31.788856 18943 tablet_service.cc:1467] Processing CreateTablet for tablet ecb8a9b04ba34f779e2227e77e6f764c (DEFAULT_TABLE table=rebalance_test_table_3 [id=210510c054044948932a30db816646aa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:31.788797 18680 tablet_service.cc:1467] Processing CreateTablet for tablet ecb8a9b04ba34f779e2227e77e6f764c (DEFAULT_TABLE table=rebalance_test_table_3 [id=210510c054044948932a30db816646aa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:31.789304 18680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecb8a9b04ba34f779e2227e77e6f764c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:31.789304 18943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecb8a9b04ba34f779e2227e77e6f764c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:31.789983 19075 tablet_service.cc:1467] Processing CreateTablet for tablet 332e4b23751b422ab7bd17b80c93701d (DEFAULT_TABLE table=rebalance_test_table_3 [id=210510c054044948932a30db816646aa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:31.790136 18812 tablet_service.cc:1467] Processing CreateTablet for tablet ecb8a9b04ba34f779e2227e77e6f764c (DEFAULT_TABLE table=rebalance_test_table_3 [id=210510c054044948932a30db816646aa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:31.790474 19075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 332e4b23751b422ab7bd17b80c93701d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:31.790557 18812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecb8a9b04ba34f779e2227e77e6f764c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:31.794548 18679 tablet_service.cc:1467] Processing CreateTablet for tablet bad03a4ec8494938839d7d6eef8c684e (DEFAULT_TABLE table=rebalance_test_table_3 [id=210510c054044948932a30db816646aa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:31.795044 18679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bad03a4ec8494938839d7d6eef8c684e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:31.795348 18945 tablet_service.cc:1467] Processing CreateTablet for tablet 332e4b23751b422ab7bd17b80c93701d (DEFAULT_TABLE table=rebalance_test_table_3 [id=210510c054044948932a30db816646aa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:31.795760 18945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 332e4b23751b422ab7bd17b80c93701d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:31.798259 18944 tablet_service.cc:1467] Processing CreateTablet for tablet bad03a4ec8494938839d7d6eef8c684e (DEFAULT_TABLE table=rebalance_test_table_3 [id=210510c054044948932a30db816646aa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:31.798576 18813 tablet_service.cc:1467] Processing CreateTablet for tablet 332e4b23751b422ab7bd17b80c93701d (DEFAULT_TABLE table=rebalance_test_table_3 [id=210510c054044948932a30db816646aa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:31.798748 18944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bad03a4ec8494938839d7d6eef8c684e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:31.798985 18813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 332e4b23751b422ab7bd17b80c93701d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:31.802604 19077 tablet_service.cc:1467] Processing CreateTablet for tablet bad03a4ec8494938839d7d6eef8c684e (DEFAULT_TABLE table=rebalance_test_table_3 [id=210510c054044948932a30db816646aa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:31.803077 19077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bad03a4ec8494938839d7d6eef8c684e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:31.813074 19969 tablet_bootstrap.cc:492] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8: Bootstrap starting.
I20240502 14:26:31.813731 19967 tablet_bootstrap.cc:492] T ecb8a9b04ba34f779e2227e77e6f764c P fec04c50dc5b49878ffa3c3873df32e4: Bootstrap starting.
I20240502 14:26:31.816064 19969 tablet_bootstrap.cc:654] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:31.816229 19967 tablet_bootstrap.cc:654] T ecb8a9b04ba34f779e2227e77e6f764c P fec04c50dc5b49878ffa3c3873df32e4: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:31.817750 19965 tablet_bootstrap.cc:492] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06: Bootstrap starting.
I20240502 14:26:31.819512 19965 tablet_bootstrap.cc:654] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:31.819516 19967 tablet_bootstrap.cc:492] T ecb8a9b04ba34f779e2227e77e6f764c P fec04c50dc5b49878ffa3c3873df32e4: No bootstrap required, opened a new log
I20240502 14:26:31.819692 19967 ts_tablet_manager.cc:1367] T ecb8a9b04ba34f779e2227e77e6f764c P fec04c50dc5b49878ffa3c3873df32e4: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240502 14:26:31.820814 19967 raft_consensus.cc:348] T ecb8a9b04ba34f779e2227e77e6f764c P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:31.821219 19967 raft_consensus.cc:374] T ecb8a9b04ba34f779e2227e77e6f764c P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:31.821403 19967 raft_consensus.cc:729] T ecb8a9b04ba34f779e2227e77e6f764c P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Initialized, Role: FOLLOWER
I20240502 14:26:31.821725 19967 consensus_queue.cc:260] T ecb8a9b04ba34f779e2227e77e6f764c P fec04c50dc5b49878ffa3c3873df32e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:31.822269 19966 tablet_bootstrap.cc:492] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b: Bootstrap starting.
I20240502 14:26:31.824823 19967 ts_tablet_manager.cc:1398] T ecb8a9b04ba34f779e2227e77e6f764c P fec04c50dc5b49878ffa3c3873df32e4: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240502 14:26:31.825017 19965 tablet_bootstrap.cc:492] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06: No bootstrap required, opened a new log
I20240502 14:26:31.825143 19967 tablet_bootstrap.cc:492] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4: Bootstrap starting.
I20240502 14:26:31.825199 19965 ts_tablet_manager.cc:1367] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240502 14:26:31.824927 19966 tablet_bootstrap.cc:654] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:31.826220 19969 tablet_bootstrap.cc:492] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8: No bootstrap required, opened a new log
I20240502 14:26:31.826172 19965 raft_consensus.cc:348] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:31.826398 19965 raft_consensus.cc:374] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:31.826498 19965 raft_consensus.cc:729] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Initialized, Role: FOLLOWER
I20240502 14:26:31.826555 19969 ts_tablet_manager.cc:1367] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.004s
I20240502 14:26:31.826812 19965 consensus_queue.cc:260] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:31.827549 19965 ts_tablet_manager.cc:1398] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:26:31.827600 19969 raft_consensus.cc:348] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:31.827857 19965 tablet_bootstrap.cc:492] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06: Bootstrap starting.
I20240502 14:26:31.827888 19969 raft_consensus.cc:374] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:31.828058 19969 raft_consensus.cc:729] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Initialized, Role: FOLLOWER
I20240502 14:26:31.828354 19969 consensus_queue.cc:260] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:31.829412 19965 tablet_bootstrap.cc:654] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:31.829834 19967 tablet_bootstrap.cc:654] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:31.830513 19969 ts_tablet_manager.cc:1398] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:26:31.830991 19969 tablet_bootstrap.cc:492] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8: Bootstrap starting.
I20240502 14:26:31.834740 19966 tablet_bootstrap.cc:492] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b: No bootstrap required, opened a new log
I20240502 14:26:31.834777 19965 tablet_bootstrap.cc:492] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06: No bootstrap required, opened a new log
I20240502 14:26:31.834932 19966 ts_tablet_manager.cc:1367] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20240502 14:26:31.835094 19965 ts_tablet_manager.cc:1367] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20240502 14:26:31.835806 19966 raft_consensus.cc:348] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:31.835848 19965 raft_consensus.cc:348] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:31.836037 19966 raft_consensus.cc:374] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:31.836124 19966 raft_consensus.cc:729] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Initialized, Role: FOLLOWER
I20240502 14:26:31.836192 19967 tablet_bootstrap.cc:492] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4: No bootstrap required, opened a new log
I20240502 14:26:31.836416 19967 ts_tablet_manager.cc:1367] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.003s
I20240502 14:26:31.836391 19966 consensus_queue.cc:260] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:31.834692 19725 raft_consensus.cc:482] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:31.836956 19966 ts_tablet_manager.cc:1398] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:26:31.836912 19965 raft_consensus.cc:374] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:31.837141 19965 raft_consensus.cc:729] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Initialized, Role: FOLLOWER
I20240502 14:26:31.837333 19967 raft_consensus.cc:348] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:31.837576 19967 raft_consensus.cc:374] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:31.837472 19965 consensus_queue.cc:260] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:31.837677 19967 raft_consensus.cc:729] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Initialized, Role: FOLLOWER
I20240502 14:26:31.837970 19967 consensus_queue.cc:260] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:31.837044 19725 raft_consensus.cc:504] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:31.839129 19725 leader_election.cc:288] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749), fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
I20240502 14:26:31.839890 19969 tablet_bootstrap.cc:654] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:31.841467 19967 ts_tablet_manager.cc:1398] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:26:31.842113 19966 tablet_bootstrap.cc:492] T bad03a4ec8494938839d7d6eef8c684e P 2db304b538e14857b7b1cc272d0db94b: Bootstrap starting.
I20240502 14:26:31.844719 18700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecb8a9b04ba34f779e2227e77e6f764c" candidate_uuid: "c902805060df4df9b5599d0e94b2dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b" is_pre_election: true
I20240502 14:26:31.845052 18700 raft_consensus.cc:2454] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c902805060df4df9b5599d0e94b2dc06 in term 0.
I20240502 14:26:31.847124 19966 tablet_bootstrap.cc:654] T bad03a4ec8494938839d7d6eef8c684e P 2db304b538e14857b7b1cc272d0db94b: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:31.848459 18898 leader_election.cc:302] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b, c902805060df4df9b5599d0e94b2dc06; no voters:
I20240502 14:26:31.851204 19725 raft_consensus.cc:2781] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:31.851397 19725 raft_consensus.cc:482] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:31.851513 19725 raft_consensus.cc:3037] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:31.852494 19966 tablet_bootstrap.cc:492] T bad03a4ec8494938839d7d6eef8c684e P 2db304b538e14857b7b1cc272d0db94b: No bootstrap required, opened a new log
I20240502 14:26:31.852727 19966 ts_tablet_manager.cc:1367] T bad03a4ec8494938839d7d6eef8c684e P 2db304b538e14857b7b1cc272d0db94b: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20240502 14:26:31.853859 19966 raft_consensus.cc:348] T bad03a4ec8494938839d7d6eef8c684e P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:31.854193 19966 raft_consensus.cc:374] T bad03a4ec8494938839d7d6eef8c684e P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:31.854276 19966 raft_consensus.cc:729] T bad03a4ec8494938839d7d6eef8c684e P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Initialized, Role: FOLLOWER
I20240502 14:26:31.854652 19966 consensus_queue.cc:260] T bad03a4ec8494938839d7d6eef8c684e P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:31.859644 19966 ts_tablet_manager.cc:1398] T bad03a4ec8494938839d7d6eef8c684e P 2db304b538e14857b7b1cc272d0db94b: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240502 14:26:31.860245 19725 raft_consensus.cc:504] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:31.862561 18700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecb8a9b04ba34f779e2227e77e6f764c" candidate_uuid: "c902805060df4df9b5599d0e94b2dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b"
I20240502 14:26:31.862903 18700 raft_consensus.cc:3037] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:31.863736 19965 ts_tablet_manager.cc:1398] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06: Time spent starting tablet: real 0.028s user 0.023s sys 0.003s
I20240502 14:26:31.864754 19965 tablet_bootstrap.cc:492] T 332e4b23751b422ab7bd17b80c93701d P c902805060df4df9b5599d0e94b2dc06: Bootstrap starting.
I20240502 14:26:31.864938 19969 tablet_bootstrap.cc:492] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8: No bootstrap required, opened a new log
I20240502 14:26:31.864792 19725 leader_election.cc:288] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 1 election: Requested vote from peers 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749), fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
I20240502 14:26:31.865118 19969 ts_tablet_manager.cc:1367] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8: Time spent bootstrapping tablet: real 0.034s user 0.009s sys 0.017s
I20240502 14:26:31.865957 19969 raft_consensus.cc:348] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:31.866186 19969 raft_consensus.cc:374] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:31.866269 19969 raft_consensus.cc:729] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Initialized, Role: FOLLOWER
I20240502 14:26:31.866572 19965 tablet_bootstrap.cc:654] T 332e4b23751b422ab7bd17b80c93701d P c902805060df4df9b5599d0e94b2dc06: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:31.866556 19969 consensus_queue.cc:260] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:31.867239 19969 ts_tablet_manager.cc:1398] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:26:31.868896 19965 tablet_bootstrap.cc:492] T 332e4b23751b422ab7bd17b80c93701d P c902805060df4df9b5599d0e94b2dc06: No bootstrap required, opened a new log
I20240502 14:26:31.869072 18700 raft_consensus.cc:2454] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c902805060df4df9b5599d0e94b2dc06 in term 1.
I20240502 14:26:31.869117 19965 ts_tablet_manager.cc:1367] T 332e4b23751b422ab7bd17b80c93701d P c902805060df4df9b5599d0e94b2dc06: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 14:26:31.869601 18898 leader_election.cc:302] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b, c902805060df4df9b5599d0e94b2dc06; no voters:
I20240502 14:26:31.869946 19725 raft_consensus.cc:2781] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:26:31.870093 19725 raft_consensus.cc:686] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Becoming Leader. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Running, Role: LEADER
I20240502 14:26:31.870244 19965 raft_consensus.cc:348] T 332e4b23751b422ab7bd17b80c93701d P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:31.870409 19725 consensus_queue.cc:237] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:31.870743 19965 raft_consensus.cc:374] T 332e4b23751b422ab7bd17b80c93701d P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:31.870895 19965 raft_consensus.cc:729] T 332e4b23751b422ab7bd17b80c93701d P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Initialized, Role: FOLLOWER
I20240502 14:26:31.871191 19965 consensus_queue.cc:260] T 332e4b23751b422ab7bd17b80c93701d P c902805060df4df9b5599d0e94b2dc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:31.871835 19965 ts_tablet_manager.cc:1398] T 332e4b23751b422ab7bd17b80c93701d P c902805060df4df9b5599d0e94b2dc06: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:26:31.874164 18552 catalog_manager.cc:5484] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 reported cstate change: term changed from 0 to 1, leader changed from <none> to c902805060df4df9b5599d0e94b2dc06 (127.5.107.67). New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } }
I20240502 14:26:31.875854 18833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecb8a9b04ba34f779e2227e77e6f764c" candidate_uuid: "c902805060df4df9b5599d0e94b2dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec04c50dc5b49878ffa3c3873df32e4"
I20240502 14:26:31.875854 18832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecb8a9b04ba34f779e2227e77e6f764c" candidate_uuid: "c902805060df4df9b5599d0e94b2dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec04c50dc5b49878ffa3c3873df32e4" is_pre_election: true
I20240502 14:26:31.876116 18833 raft_consensus.cc:3037] T ecb8a9b04ba34f779e2227e77e6f764c P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:31.878993 18833 raft_consensus.cc:2454] T ecb8a9b04ba34f779e2227e77e6f764c P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c902805060df4df9b5599d0e94b2dc06 in term 1.
I20240502 14:26:31.921923 19733 raft_consensus.cc:482] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:31.922122 19733 raft_consensus.cc:504] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:31.922861 19733 leader_election.cc:288] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749), c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:26:31.923553 18964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bad03a4ec8494938839d7d6eef8c684e" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06" is_pre_election: true
I20240502 14:26:31.923835 18964 raft_consensus.cc:2454] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 0.
I20240502 14:26:31.924284 19032 leader_election.cc:302] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 007b2e79dc424a6f80141180f072f2d8, c902805060df4df9b5599d0e94b2dc06; no voters:
I20240502 14:26:31.924636 19733 raft_consensus.cc:2781] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:31.924813 19733 raft_consensus.cc:482] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:31.924953 19733 raft_consensus.cc:3037] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:31.926328 19983 raft_consensus.cc:482] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:31.926522 19983 raft_consensus.cc:504] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:31.927386 19983 leader_election.cc:288] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409), fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
I20240502 14:26:31.927457 19733 raft_consensus.cc:504] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:31.927809 18964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "332e4b23751b422ab7bd17b80c93701d" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06" is_pre_election: true
I20240502 14:26:31.928076 18964 raft_consensus.cc:2454] T 332e4b23751b422ab7bd17b80c93701d P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 0.
I20240502 14:26:31.928102 19733 leader_election.cc:288] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 election: Requested vote from peers 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749), c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:26:31.928153 18833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "332e4b23751b422ab7bd17b80c93701d" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec04c50dc5b49878ffa3c3873df32e4" is_pre_election: true
I20240502 14:26:31.928380 18833 raft_consensus.cc:2454] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 0.
I20240502 14:26:31.928658 19032 leader_election.cc:302] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 007b2e79dc424a6f80141180f072f2d8, c902805060df4df9b5599d0e94b2dc06; no voters:
I20240502 14:26:31.929279 19733 raft_consensus.cc:2781] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:31.929337 18964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bad03a4ec8494938839d7d6eef8c684e" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06"
I20240502 14:26:31.929440 19733 raft_consensus.cc:482] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:31.929543 18964 raft_consensus.cc:3037] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:31.929636 19733 raft_consensus.cc:3037] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:31.930510 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bad03a4ec8494938839d7d6eef8c684e" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b"
I20240502 14:26:31.930473 18700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bad03a4ec8494938839d7d6eef8c684e" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b" is_pre_election: true
I20240502 14:26:31.930713 18701 raft_consensus.cc:3037] T bad03a4ec8494938839d7d6eef8c684e P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:31.932125 18964 raft_consensus.cc:2454] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 1.
I20240502 14:26:31.932157 19733 raft_consensus.cc:504] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:31.932516 19032 leader_election.cc:302] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 007b2e79dc424a6f80141180f072f2d8, c902805060df4df9b5599d0e94b2dc06; no voters:
I20240502 14:26:31.932822 19983 raft_consensus.cc:2781] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:26:31.932946 18701 raft_consensus.cc:2454] T bad03a4ec8494938839d7d6eef8c684e P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 1.
I20240502 14:26:31.933444 19983 raft_consensus.cc:686] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Becoming Leader. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Running, Role: LEADER
I20240502 14:26:31.933707 18964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "332e4b23751b422ab7bd17b80c93701d" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06"
I20240502 14:26:31.933728 18833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "332e4b23751b422ab7bd17b80c93701d" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec04c50dc5b49878ffa3c3873df32e4"
I20240502 14:26:31.933903 18833 raft_consensus.cc:3037] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:31.933897 18964 raft_consensus.cc:3037] T 332e4b23751b422ab7bd17b80c93701d P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:31.933765 19983 consensus_queue.cc:237] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:31.935920 18964 raft_consensus.cc:2454] T 332e4b23751b422ab7bd17b80c93701d P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 1.
I20240502 14:26:31.936062 18833 raft_consensus.cc:2454] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 1.
I20240502 14:26:31.936136 19733 leader_election.cc:288] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 election: Requested vote from peers c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409), fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
I20240502 14:26:31.936319 19032 leader_election.cc:302] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 007b2e79dc424a6f80141180f072f2d8, c902805060df4df9b5599d0e94b2dc06; no voters:
I20240502 14:26:31.936570 19733 raft_consensus.cc:2781] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:26:31.936700 19733 raft_consensus.cc:686] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Becoming Leader. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Running, Role: LEADER
I20240502 14:26:31.936901 19733 consensus_queue.cc:237] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:26:31.938175 18552 catalog_manager.cc:5484] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68). New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:26:31.942757 18552 catalog_manager.cc:5484] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68). New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } }
I20240502 14:26:32.310905 19725 consensus_queue.cc:1035] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:32.314958 19733 consensus_queue.cc:1035] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:32.328547 19976 consensus_queue.cc:1035] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:32.342442 19983 consensus_queue.cc:1035] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 14:26:32.368646 19983 consensus_queue.cc:1035] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:26:32.369668 19987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:26:32.370088 19987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 14:26:32.373745 19984 consensus_queue.cc:1035] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:26:32.380525 19987 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 238.89 ms
time per row: 0.119445 ms
I20240502 14:26:32.764653 18559 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48228:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:26:32.774729 18813 tablet_service.cc:1467] Processing CreateTablet for tablet b13bcd6e693f4c7687a1aabdafbb83c6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=4dd18a179f134d1babc05959227a25cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:32.775200 18813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b13bcd6e693f4c7687a1aabdafbb83c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:32.775379 18679 tablet_service.cc:1467] Processing CreateTablet for tablet 097eb705c9e24740a5be61bf3354f3d8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=4dd18a179f134d1babc05959227a25cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:32.775797 18679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 097eb705c9e24740a5be61bf3354f3d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:32.778096 18945 tablet_service.cc:1467] Processing CreateTablet for tablet b13bcd6e693f4c7687a1aabdafbb83c6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=4dd18a179f134d1babc05959227a25cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:32.778167 19075 tablet_service.cc:1467] Processing CreateTablet for tablet 097eb705c9e24740a5be61bf3354f3d8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=4dd18a179f134d1babc05959227a25cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:32.778277 19077 tablet_service.cc:1467] Processing CreateTablet for tablet 414e03d597934da3a6a74d786aeab134 (DEFAULT_TABLE table=rebalance_test_table_4 [id=4dd18a179f134d1babc05959227a25cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:32.778580 19075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 097eb705c9e24740a5be61bf3354f3d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:32.778605 18945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b13bcd6e693f4c7687a1aabdafbb83c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:32.778991 18680 tablet_service.cc:1467] Processing CreateTablet for tablet b13bcd6e693f4c7687a1aabdafbb83c6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=4dd18a179f134d1babc05959227a25cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:32.779394 18680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b13bcd6e693f4c7687a1aabdafbb83c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:32.786190 18812 tablet_service.cc:1467] Processing CreateTablet for tablet 097eb705c9e24740a5be61bf3354f3d8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=4dd18a179f134d1babc05959227a25cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:32.786303 18811 tablet_service.cc:1467] Processing CreateTablet for tablet 414e03d597934da3a6a74d786aeab134 (DEFAULT_TABLE table=rebalance_test_table_4 [id=4dd18a179f134d1babc05959227a25cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:32.786713 18812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 097eb705c9e24740a5be61bf3354f3d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:32.786924 18811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 414e03d597934da3a6a74d786aeab134. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:32.787276 19077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 414e03d597934da3a6a74d786aeab134. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:32.787281 18943 tablet_service.cc:1467] Processing CreateTablet for tablet 414e03d597934da3a6a74d786aeab134 (DEFAULT_TABLE table=rebalance_test_table_4 [id=4dd18a179f134d1babc05959227a25cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:32.787696 18943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 414e03d597934da3a6a74d786aeab134. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:32.802479 20056 tablet_bootstrap.cc:492] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06: Bootstrap starting.
I20240502 14:26:32.805616 20056 tablet_bootstrap.cc:654] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:32.805967 20055 tablet_bootstrap.cc:492] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4: Bootstrap starting.
I20240502 14:26:32.806293 20054 tablet_bootstrap.cc:492] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8: Bootstrap starting.
I20240502 14:26:32.807677 20055 tablet_bootstrap.cc:654] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:32.809075 20053 tablet_bootstrap.cc:492] T 097eb705c9e24740a5be61bf3354f3d8 P 2db304b538e14857b7b1cc272d0db94b: Bootstrap starting.
I20240502 14:26:32.809298 20054 tablet_bootstrap.cc:654] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:32.811518 20053 tablet_bootstrap.cc:654] T 097eb705c9e24740a5be61bf3354f3d8 P 2db304b538e14857b7b1cc272d0db94b: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:32.812239 20056 tablet_bootstrap.cc:492] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06: No bootstrap required, opened a new log
I20240502 14:26:32.812388 20055 tablet_bootstrap.cc:492] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4: No bootstrap required, opened a new log
I20240502 14:26:32.812484 20056 ts_tablet_manager.cc:1367] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240502 14:26:32.812537 20055 ts_tablet_manager.cc:1367] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 14:26:32.813423 20056 raft_consensus.cc:348] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:32.813768 20056 raft_consensus.cc:374] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:32.813935 20056 raft_consensus.cc:729] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Initialized, Role: FOLLOWER
I20240502 14:26:32.813843 20055 raft_consensus.cc:348] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:32.814077 20055 raft_consensus.cc:374] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:32.814173 20055 raft_consensus.cc:729] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Initialized, Role: FOLLOWER
I20240502 14:26:32.814380 20054 tablet_bootstrap.cc:492] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8: No bootstrap required, opened a new log
I20240502 14:26:32.814317 20056 consensus_queue.cc:260] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:32.814440 20055 consensus_queue.cc:260] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:32.814649 20054 ts_tablet_manager.cc:1367] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 14:26:32.815166 20056 ts_tablet_manager.cc:1398] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:26:32.815515 20056 tablet_bootstrap.cc:492] T 414e03d597934da3a6a74d786aeab134 P c902805060df4df9b5599d0e94b2dc06: Bootstrap starting.
I20240502 14:26:32.815706 20054 raft_consensus.cc:348] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:32.816272 20055 ts_tablet_manager.cc:1398] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:26:32.816577 20055 tablet_bootstrap.cc:492] T b13bcd6e693f4c7687a1aabdafbb83c6 P fec04c50dc5b49878ffa3c3873df32e4: Bootstrap starting.
I20240502 14:26:32.816999 20054 raft_consensus.cc:374] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:32.817155 20054 raft_consensus.cc:729] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Initialized, Role: FOLLOWER
I20240502 14:26:32.817515 20054 consensus_queue.cc:260] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:32.817921 20056 tablet_bootstrap.cc:654] T 414e03d597934da3a6a74d786aeab134 P c902805060df4df9b5599d0e94b2dc06: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:32.819090 20054 ts_tablet_manager.cc:1398] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:26:32.819423 20054 tablet_bootstrap.cc:492] T 414e03d597934da3a6a74d786aeab134 P 007b2e79dc424a6f80141180f072f2d8: Bootstrap starting.
I20240502 14:26:32.819792 20055 tablet_bootstrap.cc:654] T b13bcd6e693f4c7687a1aabdafbb83c6 P fec04c50dc5b49878ffa3c3873df32e4: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:32.824075 20054 tablet_bootstrap.cc:654] T 414e03d597934da3a6a74d786aeab134 P 007b2e79dc424a6f80141180f072f2d8: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:32.826778 20053 tablet_bootstrap.cc:492] T 097eb705c9e24740a5be61bf3354f3d8 P 2db304b538e14857b7b1cc272d0db94b: No bootstrap required, opened a new log
I20240502 14:26:32.826965 20053 ts_tablet_manager.cc:1367] T 097eb705c9e24740a5be61bf3354f3d8 P 2db304b538e14857b7b1cc272d0db94b: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.003s
I20240502 14:26:32.827806 20054 tablet_bootstrap.cc:492] T 414e03d597934da3a6a74d786aeab134 P 007b2e79dc424a6f80141180f072f2d8: No bootstrap required, opened a new log
I20240502 14:26:32.827862 20053 raft_consensus.cc:348] T 097eb705c9e24740a5be61bf3354f3d8 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:32.828081 20053 raft_consensus.cc:374] T 097eb705c9e24740a5be61bf3354f3d8 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:32.828022 20054 ts_tablet_manager.cc:1367] T 414e03d597934da3a6a74d786aeab134 P 007b2e79dc424a6f80141180f072f2d8: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240502 14:26:32.828168 20053 raft_consensus.cc:729] T 097eb705c9e24740a5be61bf3354f3d8 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Initialized, Role: FOLLOWER
I20240502 14:26:32.828895 20054 raft_consensus.cc:348] T 414e03d597934da3a6a74d786aeab134 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:32.829582 20054 raft_consensus.cc:374] T 414e03d597934da3a6a74d786aeab134 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:32.829711 20054 raft_consensus.cc:729] T 414e03d597934da3a6a74d786aeab134 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Initialized, Role: FOLLOWER
I20240502 14:26:32.829777 20053 consensus_queue.cc:260] T 097eb705c9e24740a5be61bf3354f3d8 P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:32.830004 20054 consensus_queue.cc:260] T 414e03d597934da3a6a74d786aeab134 P 007b2e79dc424a6f80141180f072f2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:32.830379 20053 ts_tablet_manager.cc:1398] T 097eb705c9e24740a5be61bf3354f3d8 P 2db304b538e14857b7b1cc272d0db94b: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240502 14:26:32.830670 20053 tablet_bootstrap.cc:492] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b: Bootstrap starting.
I20240502 14:26:32.831280 20054 ts_tablet_manager.cc:1398] T 414e03d597934da3a6a74d786aeab134 P 007b2e79dc424a6f80141180f072f2d8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:26:32.833460 20056 tablet_bootstrap.cc:492] T 414e03d597934da3a6a74d786aeab134 P c902805060df4df9b5599d0e94b2dc06: No bootstrap required, opened a new log
I20240502 14:26:32.833693 20056 ts_tablet_manager.cc:1367] T 414e03d597934da3a6a74d786aeab134 P c902805060df4df9b5599d0e94b2dc06: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.000s
I20240502 14:26:32.834508 20055 tablet_bootstrap.cc:492] T b13bcd6e693f4c7687a1aabdafbb83c6 P fec04c50dc5b49878ffa3c3873df32e4: No bootstrap required, opened a new log
I20240502 14:26:32.834725 20055 ts_tablet_manager.cc:1367] T b13bcd6e693f4c7687a1aabdafbb83c6 P fec04c50dc5b49878ffa3c3873df32e4: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.014s
I20240502 14:26:32.834656 20056 raft_consensus.cc:348] T 414e03d597934da3a6a74d786aeab134 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:32.835013 20056 raft_consensus.cc:374] T 414e03d597934da3a6a74d786aeab134 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:32.835129 20056 raft_consensus.cc:729] T 414e03d597934da3a6a74d786aeab134 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Initialized, Role: FOLLOWER
I20240502 14:26:32.835479 20056 consensus_queue.cc:260] T 414e03d597934da3a6a74d786aeab134 P c902805060df4df9b5599d0e94b2dc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:32.835605 20055 raft_consensus.cc:348] T b13bcd6e693f4c7687a1aabdafbb83c6 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:32.835824 20055 raft_consensus.cc:374] T b13bcd6e693f4c7687a1aabdafbb83c6 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:32.835899 20055 raft_consensus.cc:729] T b13bcd6e693f4c7687a1aabdafbb83c6 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Initialized, Role: FOLLOWER
I20240502 14:26:32.836017 20053 tablet_bootstrap.cc:654] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:32.836078 20056 ts_tablet_manager.cc:1398] T 414e03d597934da3a6a74d786aeab134 P c902805060df4df9b5599d0e94b2dc06: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:26:32.836184 20055 consensus_queue.cc:260] T b13bcd6e693f4c7687a1aabdafbb83c6 P fec04c50dc5b49878ffa3c3873df32e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:32.836745 20055 ts_tablet_manager.cc:1398] T b13bcd6e693f4c7687a1aabdafbb83c6 P fec04c50dc5b49878ffa3c3873df32e4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:26:32.837428 20055 tablet_bootstrap.cc:492] T 097eb705c9e24740a5be61bf3354f3d8 P fec04c50dc5b49878ffa3c3873df32e4: Bootstrap starting.
I20240502 14:26:32.838698 20053 tablet_bootstrap.cc:492] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b: No bootstrap required, opened a new log
I20240502 14:26:32.838939 20053 ts_tablet_manager.cc:1367] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240502 14:26:32.839088 20055 tablet_bootstrap.cc:654] T 097eb705c9e24740a5be61bf3354f3d8 P fec04c50dc5b49878ffa3c3873df32e4: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:32.839902 20053 raft_consensus.cc:348] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:32.840971 20053 raft_consensus.cc:374] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:32.841059 20053 raft_consensus.cc:729] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Initialized, Role: FOLLOWER
I20240502 14:26:32.841356 20053 consensus_queue.cc:260] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:32.841953 20053 ts_tablet_manager.cc:1398] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:26:32.843312 20055 tablet_bootstrap.cc:492] T 097eb705c9e24740a5be61bf3354f3d8 P fec04c50dc5b49878ffa3c3873df32e4: No bootstrap required, opened a new log
I20240502 14:26:32.843458 20055 ts_tablet_manager.cc:1367] T 097eb705c9e24740a5be61bf3354f3d8 P fec04c50dc5b49878ffa3c3873df32e4: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20240502 14:26:32.844074 20055 raft_consensus.cc:348] T 097eb705c9e24740a5be61bf3354f3d8 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:32.844249 20055 raft_consensus.cc:374] T 097eb705c9e24740a5be61bf3354f3d8 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:32.844345 20055 raft_consensus.cc:729] T 097eb705c9e24740a5be61bf3354f3d8 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Initialized, Role: FOLLOWER
I20240502 14:26:32.844558 20055 consensus_queue.cc:260] T 097eb705c9e24740a5be61bf3354f3d8 P fec04c50dc5b49878ffa3c3873df32e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:32.845016 20055 ts_tablet_manager.cc:1398] T 097eb705c9e24740a5be61bf3354f3d8 P fec04c50dc5b49878ffa3c3873df32e4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:26:32.946152 19954 raft_consensus.cc:482] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:32.946360 19954 raft_consensus.cc:504] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:32.947012 19954 leader_election.cc:288] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463), c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:26:32.947482 19096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414e03d597934da3a6a74d786aeab134" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b2e79dc424a6f80141180f072f2d8" is_pre_election: true
I20240502 14:26:32.947566 18964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414e03d597934da3a6a74d786aeab134" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06" is_pre_election: true
I20240502 14:26:32.947721 19096 raft_consensus.cc:2454] T 414e03d597934da3a6a74d786aeab134 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 0.
I20240502 14:26:32.947811 18964 raft_consensus.cc:2454] T 414e03d597934da3a6a74d786aeab134 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 0.
I20240502 14:26:32.948150 18769 leader_election.cc:302] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 007b2e79dc424a6f80141180f072f2d8, fec04c50dc5b49878ffa3c3873df32e4; no voters:
I20240502 14:26:32.948403 19954 raft_consensus.cc:2781] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:32.948506 19954 raft_consensus.cc:482] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:32.948594 19954 raft_consensus.cc:3037] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:32.950546 19954 raft_consensus.cc:504] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:32.951220 19954 leader_election.cc:288] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 election: Requested vote from peers 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463), c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:26:32.951709 19096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414e03d597934da3a6a74d786aeab134" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b2e79dc424a6f80141180f072f2d8"
I20240502 14:26:32.951727 18964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414e03d597934da3a6a74d786aeab134" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06"
I20240502 14:26:32.951933 18964 raft_consensus.cc:3037] T 414e03d597934da3a6a74d786aeab134 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:32.951930 19096 raft_consensus.cc:3037] T 414e03d597934da3a6a74d786aeab134 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:32.953994 19096 raft_consensus.cc:2454] T 414e03d597934da3a6a74d786aeab134 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 1.
I20240502 14:26:32.954088 18964 raft_consensus.cc:2454] T 414e03d597934da3a6a74d786aeab134 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 1.
I20240502 14:26:32.954429 18769 leader_election.cc:302] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 007b2e79dc424a6f80141180f072f2d8, fec04c50dc5b49878ffa3c3873df32e4; no voters:
I20240502 14:26:32.954715 19954 raft_consensus.cc:2781] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:26:32.955075 19954 raft_consensus.cc:686] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Becoming Leader. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Running, Role: LEADER
I20240502 14:26:32.955394 19954 consensus_queue.cc:237] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:32.958035 18551 catalog_manager.cc:5484] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66). New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
I20240502 14:26:32.991715 19984 raft_consensus.cc:482] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:32.991951 19984 raft_consensus.cc:504] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:32.992683 19984 leader_election.cc:288] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009), 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:26:32.993367 18832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "097eb705c9e24740a5be61bf3354f3d8" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec04c50dc5b49878ffa3c3873df32e4" is_pre_election: true
I20240502 14:26:32.993384 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "097eb705c9e24740a5be61bf3354f3d8" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b" is_pre_election: true
I20240502 14:26:32.993664 18832 raft_consensus.cc:2454] T 097eb705c9e24740a5be61bf3354f3d8 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 0.
I20240502 14:26:32.993659 18701 raft_consensus.cc:2454] T 097eb705c9e24740a5be61bf3354f3d8 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 0.
I20240502 14:26:32.994177 19029 leader_election.cc:302] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 007b2e79dc424a6f80141180f072f2d8, fec04c50dc5b49878ffa3c3873df32e4; no voters:
I20240502 14:26:32.994474 19984 raft_consensus.cc:2781] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:32.994556 19984 raft_consensus.cc:482] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:32.994668 19984 raft_consensus.cc:3037] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:32.996510 19984 raft_consensus.cc:504] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:32.997057 19984 leader_election.cc:288] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 election: Requested vote from peers fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009), 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:26:32.997588 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "097eb705c9e24740a5be61bf3354f3d8" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b"
I20240502 14:26:32.997650 18832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "097eb705c9e24740a5be61bf3354f3d8" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec04c50dc5b49878ffa3c3873df32e4"
I20240502 14:26:32.997808 18701 raft_consensus.cc:3037] T 097eb705c9e24740a5be61bf3354f3d8 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:32.997839 18832 raft_consensus.cc:3037] T 097eb705c9e24740a5be61bf3354f3d8 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:32.999747 18701 raft_consensus.cc:2454] T 097eb705c9e24740a5be61bf3354f3d8 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 1.
I20240502 14:26:32.999747 18832 raft_consensus.cc:2454] T 097eb705c9e24740a5be61bf3354f3d8 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 1.
I20240502 14:26:33.000236 19030 leader_election.cc:302] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 007b2e79dc424a6f80141180f072f2d8, 2db304b538e14857b7b1cc272d0db94b, fec04c50dc5b49878ffa3c3873df32e4; no voters:
I20240502 14:26:33.000609 19984 raft_consensus.cc:2781] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:26:33.000764 19984 raft_consensus.cc:686] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Becoming Leader. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Running, Role: LEADER
I20240502 14:26:33.001044 19984 consensus_queue.cc:237] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:33.003859 18551 catalog_manager.cc:5484] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68). New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:26:33.026335 19976 raft_consensus.cc:482] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:33.026577 19976 raft_consensus.cc:504] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:33.027247 19976 leader_election.cc:288] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009), 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:26:33.027805 18832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13bcd6e693f4c7687a1aabdafbb83c6" candidate_uuid: "c902805060df4df9b5599d0e94b2dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec04c50dc5b49878ffa3c3873df32e4" is_pre_election: true
I20240502 14:26:33.028056 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13bcd6e693f4c7687a1aabdafbb83c6" candidate_uuid: "c902805060df4df9b5599d0e94b2dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b" is_pre_election: true
I20240502 14:26:33.028276 18701 raft_consensus.cc:2454] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c902805060df4df9b5599d0e94b2dc06 in term 0.
I20240502 14:26:33.028435 18832 raft_consensus.cc:2454] T b13bcd6e693f4c7687a1aabdafbb83c6 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c902805060df4df9b5599d0e94b2dc06 in term 0.
I20240502 14:26:33.028728 18898 leader_election.cc:302] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b, c902805060df4df9b5599d0e94b2dc06; no voters:
I20240502 14:26:33.029009 19976 raft_consensus.cc:2781] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:33.029090 19976 raft_consensus.cc:482] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:33.029220 19976 raft_consensus.cc:3037] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:33.031081 19976 raft_consensus.cc:504] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:33.031818 19976 leader_election.cc:288] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 1 election: Requested vote from peers fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009), 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:26:33.032303 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13bcd6e693f4c7687a1aabdafbb83c6" candidate_uuid: "c902805060df4df9b5599d0e94b2dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b"
I20240502 14:26:33.032325 18832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13bcd6e693f4c7687a1aabdafbb83c6" candidate_uuid: "c902805060df4df9b5599d0e94b2dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec04c50dc5b49878ffa3c3873df32e4"
I20240502 14:26:33.032501 18701 raft_consensus.cc:3037] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:33.032536 18832 raft_consensus.cc:3037] T b13bcd6e693f4c7687a1aabdafbb83c6 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:33.034415 18701 raft_consensus.cc:2454] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c902805060df4df9b5599d0e94b2dc06 in term 1.
I20240502 14:26:33.034545 18832 raft_consensus.cc:2454] T b13bcd6e693f4c7687a1aabdafbb83c6 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c902805060df4df9b5599d0e94b2dc06 in term 1.
I20240502 14:26:33.034838 18898 leader_election.cc:302] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b, c902805060df4df9b5599d0e94b2dc06; no voters:
I20240502 14:26:33.035141 19976 raft_consensus.cc:2781] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:26:33.035308 19976 raft_consensus.cc:686] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Becoming Leader. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Running, Role: LEADER
I20240502 14:26:33.035573 19976 consensus_queue.cc:237] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:33.038190 18551 catalog_manager.cc:5484] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 reported cstate change: term changed from 0 to 1, leader changed from <none> to c902805060df4df9b5599d0e94b2dc06 (127.5.107.67). New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
I20240502 14:26:33.436956 19976 consensus_queue.cc:1035] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:26:33.442245 20070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:26:33.442821 20070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 14:26:33.443275 19725 consensus_queue.cc:1035] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:26:33.452826 20070 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:26:33.482497 19984 consensus_queue.cc:1035] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:33.490097 20038 consensus_queue.cc:1035] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240502 14:26:33.510754 19954 consensus_queue.cc:1035] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:33.518400 20097 consensus_queue.cc:1035] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 273.533 ms
time per row: 0.136766 ms
I20240502 14:26:33.899694 18551 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48228:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:26:33.911046 18680 tablet_service.cc:1467] Processing CreateTablet for tablet 6a992d12aee7416096d1353533397386 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ff5cff7e35c746a5bc64cc06f18bf7d9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:33.911365 18679 tablet_service.cc:1467] Processing CreateTablet for tablet b15d11209e45473e863766e5cd6e822f (DEFAULT_TABLE table=rebalance_test_table_5 [id=ff5cff7e35c746a5bc64cc06f18bf7d9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:33.911358 18811 tablet_service.cc:1467] Processing CreateTablet for tablet 67827c0262724897a05bf80df579bbf3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ff5cff7e35c746a5bc64cc06f18bf7d9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:33.911515 18680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a992d12aee7416096d1353533397386. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:33.911753 18811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67827c0262724897a05bf80df579bbf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:33.911757 18679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b15d11209e45473e863766e5cd6e822f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:33.914444 18945 tablet_service.cc:1467] Processing CreateTablet for tablet b15d11209e45473e863766e5cd6e822f (DEFAULT_TABLE table=rebalance_test_table_5 [id=ff5cff7e35c746a5bc64cc06f18bf7d9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:33.914667 19077 tablet_service.cc:1467] Processing CreateTablet for tablet 6a992d12aee7416096d1353533397386 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ff5cff7e35c746a5bc64cc06f18bf7d9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:33.914889 18945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b15d11209e45473e863766e5cd6e822f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:33.915083 19077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a992d12aee7416096d1353533397386. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:33.917559 18943 tablet_service.cc:1467] Processing CreateTablet for tablet 6a992d12aee7416096d1353533397386 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ff5cff7e35c746a5bc64cc06f18bf7d9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:33.917582 19075 tablet_service.cc:1467] Processing CreateTablet for tablet b15d11209e45473e863766e5cd6e822f (DEFAULT_TABLE table=rebalance_test_table_5 [id=ff5cff7e35c746a5bc64cc06f18bf7d9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:33.917927 18943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a992d12aee7416096d1353533397386. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:33.917960 19075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b15d11209e45473e863766e5cd6e822f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:33.921435 18944 tablet_service.cc:1467] Processing CreateTablet for tablet 67827c0262724897a05bf80df579bbf3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ff5cff7e35c746a5bc64cc06f18bf7d9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:33.921861 18944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67827c0262724897a05bf80df579bbf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:33.925968 18681 tablet_service.cc:1467] Processing CreateTablet for tablet 67827c0262724897a05bf80df579bbf3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ff5cff7e35c746a5bc64cc06f18bf7d9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:33.926504 18681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67827c0262724897a05bf80df579bbf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:33.927722 20138 tablet_bootstrap.cc:492] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b: Bootstrap starting.
I20240502 14:26:33.930073 20138 tablet_bootstrap.cc:654] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:33.938380 20138 tablet_bootstrap.cc:492] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b: No bootstrap required, opened a new log
I20240502 14:26:33.938441 20139 tablet_bootstrap.cc:492] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8: Bootstrap starting.
I20240502 14:26:33.938565 20138 ts_tablet_manager.cc:1367] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240502 14:26:33.939363 20138 raft_consensus.cc:348] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:33.939599 20138 raft_consensus.cc:374] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:33.939692 20138 raft_consensus.cc:729] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Initialized, Role: FOLLOWER
I20240502 14:26:33.939946 20138 consensus_queue.cc:260] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:33.940183 20139 tablet_bootstrap.cc:654] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:33.943336 20138 ts_tablet_manager.cc:1398] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 14:26:33.945962 20138 tablet_bootstrap.cc:492] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b: Bootstrap starting.
I20240502 14:26:33.948709 20138 tablet_bootstrap.cc:654] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:33.949208 20142 tablet_bootstrap.cc:492] T 6a992d12aee7416096d1353533397386 P c902805060df4df9b5599d0e94b2dc06: Bootstrap starting.
I20240502 14:26:33.951259 20139 tablet_bootstrap.cc:492] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8: No bootstrap required, opened a new log
I20240502 14:26:33.951628 20140 tablet_bootstrap.cc:492] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4: Bootstrap starting.
I20240502 14:26:33.952039 20139 ts_tablet_manager.cc:1367] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240502 14:26:33.952884 20139 raft_consensus.cc:348] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:33.953168 20139 raft_consensus.cc:374] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:33.953317 20139 raft_consensus.cc:729] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Initialized, Role: FOLLOWER
I20240502 14:26:33.953408 20138 tablet_bootstrap.cc:492] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b: No bootstrap required, opened a new log
I20240502 14:26:33.953584 20138 ts_tablet_manager.cc:1367] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.001s
I20240502 14:26:33.953745 20142 tablet_bootstrap.cc:654] T 6a992d12aee7416096d1353533397386 P c902805060df4df9b5599d0e94b2dc06: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:33.953699 20139 consensus_queue.cc:260] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:33.954465 20139 ts_tablet_manager.cc:1398] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:26:33.954429 20138 raft_consensus.cc:348] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:33.954659 20138 raft_consensus.cc:374] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:33.954749 20138 raft_consensus.cc:729] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Initialized, Role: FOLLOWER
I20240502 14:26:33.954967 20139 tablet_bootstrap.cc:492] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8: Bootstrap starting.
I20240502 14:26:33.955036 20138 consensus_queue.cc:260] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:33.955696 20138 ts_tablet_manager.cc:1398] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:26:33.956017 20138 tablet_bootstrap.cc:492] T 67827c0262724897a05bf80df579bbf3 P 2db304b538e14857b7b1cc272d0db94b: Bootstrap starting.
I20240502 14:26:33.956395 20140 tablet_bootstrap.cc:654] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:33.956913 20139 tablet_bootstrap.cc:654] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:33.960093 20142 tablet_bootstrap.cc:492] T 6a992d12aee7416096d1353533397386 P c902805060df4df9b5599d0e94b2dc06: No bootstrap required, opened a new log
I20240502 14:26:33.960204 20138 tablet_bootstrap.cc:654] T 67827c0262724897a05bf80df579bbf3 P 2db304b538e14857b7b1cc272d0db94b: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:33.960376 20142 ts_tablet_manager.cc:1367] T 6a992d12aee7416096d1353533397386 P c902805060df4df9b5599d0e94b2dc06: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.005s
I20240502 14:26:33.961468 20142 raft_consensus.cc:348] T 6a992d12aee7416096d1353533397386 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:33.961865 20142 raft_consensus.cc:374] T 6a992d12aee7416096d1353533397386 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:33.962039 20142 raft_consensus.cc:729] T 6a992d12aee7416096d1353533397386 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Initialized, Role: FOLLOWER
I20240502 14:26:33.962437 20142 consensus_queue.cc:260] T 6a992d12aee7416096d1353533397386 P c902805060df4df9b5599d0e94b2dc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:33.966991 20142 ts_tablet_manager.cc:1398] T 6a992d12aee7416096d1353533397386 P c902805060df4df9b5599d0e94b2dc06: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:26:33.970783 20142 tablet_bootstrap.cc:492] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06: Bootstrap starting.
I20240502 14:26:33.973270 20139 tablet_bootstrap.cc:492] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8: No bootstrap required, opened a new log
I20240502 14:26:33.973546 20139 ts_tablet_manager.cc:1367] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.004s
I20240502 14:26:33.973815 20140 tablet_bootstrap.cc:492] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4: No bootstrap required, opened a new log
I20240502 14:26:33.974011 20140 ts_tablet_manager.cc:1367] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.000s
I20240502 14:26:33.974786 20139 raft_consensus.cc:348] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:33.975169 20139 raft_consensus.cc:374] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:33.975152 20140 raft_consensus.cc:348] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:33.975304 20139 raft_consensus.cc:729] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Initialized, Role: FOLLOWER
I20240502 14:26:33.975433 20140 raft_consensus.cc:374] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:33.975529 20140 raft_consensus.cc:729] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Initialized, Role: FOLLOWER
I20240502 14:26:33.975750 20140 consensus_queue.cc:260] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:33.976187 20139 consensus_queue.cc:260] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:33.976397 20140 ts_tablet_manager.cc:1398] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:26:33.976771 20142 tablet_bootstrap.cc:654] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:33.978134 20139 ts_tablet_manager.cc:1398] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:26:33.980991 20138 tablet_bootstrap.cc:492] T 67827c0262724897a05bf80df579bbf3 P 2db304b538e14857b7b1cc272d0db94b: No bootstrap required, opened a new log
I20240502 14:26:33.981173 20138 ts_tablet_manager.cc:1367] T 67827c0262724897a05bf80df579bbf3 P 2db304b538e14857b7b1cc272d0db94b: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.005s
I20240502 14:26:33.982084 20138 raft_consensus.cc:348] T 67827c0262724897a05bf80df579bbf3 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:33.982306 20138 raft_consensus.cc:374] T 67827c0262724897a05bf80df579bbf3 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:33.982426 20138 raft_consensus.cc:729] T 67827c0262724897a05bf80df579bbf3 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Initialized, Role: FOLLOWER
I20240502 14:26:33.982664 20138 consensus_queue.cc:260] T 67827c0262724897a05bf80df579bbf3 P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:33.983295 20138 ts_tablet_manager.cc:1398] T 67827c0262724897a05bf80df579bbf3 P 2db304b538e14857b7b1cc272d0db94b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:26:33.987278 20142 tablet_bootstrap.cc:492] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06: No bootstrap required, opened a new log
I20240502 14:26:33.987471 20142 ts_tablet_manager.cc:1367] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.003s
I20240502 14:26:33.988415 20142 raft_consensus.cc:348] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:33.988649 20142 raft_consensus.cc:374] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:33.988729 20142 raft_consensus.cc:729] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Initialized, Role: FOLLOWER
I20240502 14:26:33.988976 20142 consensus_queue.cc:260] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:33.989543 20142 ts_tablet_manager.cc:1398] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:26:33.989835 20142 tablet_bootstrap.cc:492] T 67827c0262724897a05bf80df579bbf3 P c902805060df4df9b5599d0e94b2dc06: Bootstrap starting.
I20240502 14:26:33.991412 20142 tablet_bootstrap.cc:654] T 67827c0262724897a05bf80df579bbf3 P c902805060df4df9b5599d0e94b2dc06: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:33.993505 20142 tablet_bootstrap.cc:492] T 67827c0262724897a05bf80df579bbf3 P c902805060df4df9b5599d0e94b2dc06: No bootstrap required, opened a new log
I20240502 14:26:33.993651 20142 ts_tablet_manager.cc:1367] T 67827c0262724897a05bf80df579bbf3 P c902805060df4df9b5599d0e94b2dc06: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240502 14:26:33.994362 20142 raft_consensus.cc:348] T 67827c0262724897a05bf80df579bbf3 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:33.994570 20142 raft_consensus.cc:374] T 67827c0262724897a05bf80df579bbf3 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:33.994643 20142 raft_consensus.cc:729] T 67827c0262724897a05bf80df579bbf3 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Initialized, Role: FOLLOWER
I20240502 14:26:33.994834 20142 consensus_queue.cc:260] T 67827c0262724897a05bf80df579bbf3 P c902805060df4df9b5599d0e94b2dc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:33.995291 20142 ts_tablet_manager.cc:1398] T 67827c0262724897a05bf80df579bbf3 P c902805060df4df9b5599d0e94b2dc06: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:26:34.096565 19984 raft_consensus.cc:482] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:34.096807 19984 raft_consensus.cc:504] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:34.097470 19984 leader_election.cc:288] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409), 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:26:34.098062 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a992d12aee7416096d1353533397386" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b" is_pre_election: true
I20240502 14:26:34.098093 18964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a992d12aee7416096d1353533397386" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06" is_pre_election: true
I20240502 14:26:34.098291 18701 raft_consensus.cc:2454] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 0.
I20240502 14:26:34.098310 18964 raft_consensus.cc:2454] T 6a992d12aee7416096d1353533397386 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 0.
I20240502 14:26:34.098759 19030 leader_election.cc:302] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 007b2e79dc424a6f80141180f072f2d8, 2db304b538e14857b7b1cc272d0db94b, c902805060df4df9b5599d0e94b2dc06; no voters:
I20240502 14:26:34.099032 19984 raft_consensus.cc:2781] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:34.099120 19984 raft_consensus.cc:482] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:34.099210 19984 raft_consensus.cc:3037] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:34.101022 19984 raft_consensus.cc:504] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:34.101716 19984 leader_election.cc:288] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 election: Requested vote from peers c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409), 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:26:34.102139 18964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a992d12aee7416096d1353533397386" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06"
I20240502 14:26:34.102231 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a992d12aee7416096d1353533397386" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b"
I20240502 14:26:34.102392 18964 raft_consensus.cc:3037] T 6a992d12aee7416096d1353533397386 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:34.102480 18701 raft_consensus.cc:3037] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:34.105259 18701 raft_consensus.cc:2454] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 1.
I20240502 14:26:34.105273 18964 raft_consensus.cc:2454] T 6a992d12aee7416096d1353533397386 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 1.
I20240502 14:26:34.105913 19030 leader_election.cc:302] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 007b2e79dc424a6f80141180f072f2d8, 2db304b538e14857b7b1cc272d0db94b; no voters:
I20240502 14:26:34.106243 19984 raft_consensus.cc:2781] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:26:34.106427 19984 raft_consensus.cc:686] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Becoming Leader. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Running, Role: LEADER
I20240502 14:26:34.106714 19984 consensus_queue.cc:237] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:34.109494 18560 catalog_manager.cc:5484] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68). New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:26:34.187729 19984 raft_consensus.cc:482] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:34.187978 19984 raft_consensus.cc:504] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:34.188658 19984 leader_election.cc:288] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749), c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:26:34.189225 18964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15d11209e45473e863766e5cd6e822f" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06" is_pre_election: true
I20240502 14:26:34.189247 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15d11209e45473e863766e5cd6e822f" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b" is_pre_election: true
I20240502 14:26:34.189489 18964 raft_consensus.cc:2454] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 0.
I20240502 14:26:34.189517 18701 raft_consensus.cc:2454] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 0.
I20240502 14:26:34.189905 19032 leader_election.cc:302] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 007b2e79dc424a6f80141180f072f2d8, c902805060df4df9b5599d0e94b2dc06; no voters:
I20240502 14:26:34.190163 19984 raft_consensus.cc:2781] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:34.190261 19984 raft_consensus.cc:482] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:34.190346 19984 raft_consensus.cc:3037] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:34.192150 19984 raft_consensus.cc:504] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:34.192741 19984 leader_election.cc:288] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 election: Requested vote from peers 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749), c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:26:34.193289 18964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15d11209e45473e863766e5cd6e822f" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06"
I20240502 14:26:34.193305 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15d11209e45473e863766e5cd6e822f" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b"
I20240502 14:26:34.193513 18701 raft_consensus.cc:3037] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:34.193528 18964 raft_consensus.cc:3037] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:34.196265 18964 raft_consensus.cc:2454] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 1.
I20240502 14:26:34.196280 18701 raft_consensus.cc:2454] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 1.
I20240502 14:26:34.196707 19030 leader_election.cc:302] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 007b2e79dc424a6f80141180f072f2d8, 2db304b538e14857b7b1cc272d0db94b; no voters:
I20240502 14:26:34.196990 19984 raft_consensus.cc:2781] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:26:34.197115 19984 raft_consensus.cc:686] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Becoming Leader. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Running, Role: LEADER
I20240502 14:26:34.197391 19984 consensus_queue.cc:237] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:34.197876 20097 raft_consensus.cc:482] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:34.198065 20097 raft_consensus.cc:504] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:34.198745 20097 leader_election.cc:288] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409), 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:26:34.199365 18964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67827c0262724897a05bf80df579bbf3" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06" is_pre_election: true
I20240502 14:26:34.199563 18964 raft_consensus.cc:2454] T 67827c0262724897a05bf80df579bbf3 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 0.
I20240502 14:26:34.199976 18769 leader_election.cc:302] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c902805060df4df9b5599d0e94b2dc06, fec04c50dc5b49878ffa3c3873df32e4; no voters:
I20240502 14:26:34.199934 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67827c0262724897a05bf80df579bbf3" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b" is_pre_election: true
I20240502 14:26:34.200147 18701 raft_consensus.cc:2454] T 67827c0262724897a05bf80df579bbf3 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 0.
I20240502 14:26:34.200243 20097 raft_consensus.cc:2781] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:34.200340 20097 raft_consensus.cc:482] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:34.200469 20097 raft_consensus.cc:3037] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:34.200631 18560 catalog_manager.cc:5484] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68). New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:26:34.202775 20097 raft_consensus.cc:504] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:34.203884 18964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67827c0262724897a05bf80df579bbf3" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06"
I20240502 14:26:34.203981 20097 leader_election.cc:288] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 election: Requested vote from peers c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409), 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:26:34.204113 18964 raft_consensus.cc:3037] T 67827c0262724897a05bf80df579bbf3 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:34.204437 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67827c0262724897a05bf80df579bbf3" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b"
I20240502 14:26:34.204700 18701 raft_consensus.cc:3037] T 67827c0262724897a05bf80df579bbf3 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:34.206943 18964 raft_consensus.cc:2454] T 67827c0262724897a05bf80df579bbf3 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 1.
I20240502 14:26:34.207342 18701 raft_consensus.cc:2454] T 67827c0262724897a05bf80df579bbf3 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 1.
I20240502 14:26:34.207347 18769 leader_election.cc:302] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c902805060df4df9b5599d0e94b2dc06, fec04c50dc5b49878ffa3c3873df32e4; no voters:
I20240502 14:26:34.207676 20097 raft_consensus.cc:2781] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:26:34.207844 20097 raft_consensus.cc:686] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Becoming Leader. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Running, Role: LEADER
I20240502 14:26:34.208083 20097 consensus_queue.cc:237] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:34.210842 18560 catalog_manager.cc:5484] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66). New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
W20240502 14:26:34.582304 20154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:26:34.582686 20154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:26:34.590939 20154 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240502 14:26:34.649075 19984 consensus_queue.cc:1035] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:34.685976 19984 consensus_queue.cc:1035] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:34.690032 19984 consensus_queue.cc:1035] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:34.696982 20098 consensus_queue.cc:1035] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:34.721077 19984 consensus_queue.cc:1035] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:34.721385 20098 consensus_queue.cc:1035] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 291.511 ms
time per row: 0.145755 ms
I20240502 14:26:35.043896 18553 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48228:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:26:35.053483 18811 tablet_service.cc:1467] Processing CreateTablet for tablet b315327141954b53a0f87c2d1dc14b9b (DEFAULT_TABLE table=rebalance_test_table_6 [id=995bc008b9f94332bfa2c42a77e41bb2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:35.053480 19077 tablet_service.cc:1467] Processing CreateTablet for tablet b315327141954b53a0f87c2d1dc14b9b (DEFAULT_TABLE table=rebalance_test_table_6 [id=995bc008b9f94332bfa2c42a77e41bb2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:35.053982 19077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b315327141954b53a0f87c2d1dc14b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:35.053982 18811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b315327141954b53a0f87c2d1dc14b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:35.055159 18943 tablet_service.cc:1467] Processing CreateTablet for tablet 8a2bed29fcd64f7bbb04286404ef09de (DEFAULT_TABLE table=rebalance_test_table_6 [id=995bc008b9f94332bfa2c42a77e41bb2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:35.055486 19076 tablet_service.cc:1467] Processing CreateTablet for tablet e5fb3b049ba94af998791640bd0e31c5 (DEFAULT_TABLE table=rebalance_test_table_6 [id=995bc008b9f94332bfa2c42a77e41bb2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:35.055620 18943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a2bed29fcd64f7bbb04286404ef09de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:35.055881 19076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5fb3b049ba94af998791640bd0e31c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:35.057988 18813 tablet_service.cc:1467] Processing CreateTablet for tablet e5fb3b049ba94af998791640bd0e31c5 (DEFAULT_TABLE table=rebalance_test_table_6 [id=995bc008b9f94332bfa2c42a77e41bb2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:35.058378 18813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5fb3b049ba94af998791640bd0e31c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:35.060281 18812 tablet_service.cc:1467] Processing CreateTablet for tablet 8a2bed29fcd64f7bbb04286404ef09de (DEFAULT_TABLE table=rebalance_test_table_6 [id=995bc008b9f94332bfa2c42a77e41bb2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:35.060451 18681 tablet_service.cc:1467] Processing CreateTablet for tablet b315327141954b53a0f87c2d1dc14b9b (DEFAULT_TABLE table=rebalance_test_table_6 [id=995bc008b9f94332bfa2c42a77e41bb2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:35.060703 18812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a2bed29fcd64f7bbb04286404ef09de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:35.060870 18681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b315327141954b53a0f87c2d1dc14b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:35.061618 19075 tablet_service.cc:1467] Processing CreateTablet for tablet 8a2bed29fcd64f7bbb04286404ef09de (DEFAULT_TABLE table=rebalance_test_table_6 [id=995bc008b9f94332bfa2c42a77e41bb2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:35.062024 19075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a2bed29fcd64f7bbb04286404ef09de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:35.065659 18680 tablet_service.cc:1467] Processing CreateTablet for tablet e5fb3b049ba94af998791640bd0e31c5 (DEFAULT_TABLE table=rebalance_test_table_6 [id=995bc008b9f94332bfa2c42a77e41bb2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:35.065840 20215 tablet_bootstrap.cc:492] T b315327141954b53a0f87c2d1dc14b9b P fec04c50dc5b49878ffa3c3873df32e4: Bootstrap starting.
I20240502 14:26:35.066059 18680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5fb3b049ba94af998791640bd0e31c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:35.069904 20215 tablet_bootstrap.cc:654] T b315327141954b53a0f87c2d1dc14b9b P fec04c50dc5b49878ffa3c3873df32e4: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:35.078317 20218 tablet_bootstrap.cc:492] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b: Bootstrap starting.
I20240502 14:26:35.078812 20216 tablet_bootstrap.cc:492] T 8a2bed29fcd64f7bbb04286404ef09de P c902805060df4df9b5599d0e94b2dc06: Bootstrap starting.
I20240502 14:26:35.079509 20215 tablet_bootstrap.cc:492] T b315327141954b53a0f87c2d1dc14b9b P fec04c50dc5b49878ffa3c3873df32e4: No bootstrap required, opened a new log
I20240502 14:26:35.079664 20215 ts_tablet_manager.cc:1367] T b315327141954b53a0f87c2d1dc14b9b P fec04c50dc5b49878ffa3c3873df32e4: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20240502 14:26:35.080332 20215 raft_consensus.cc:348] T b315327141954b53a0f87c2d1dc14b9b P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:35.080605 20215 raft_consensus.cc:374] T b315327141954b53a0f87c2d1dc14b9b P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:35.080678 20216 tablet_bootstrap.cc:654] T 8a2bed29fcd64f7bbb04286404ef09de P c902805060df4df9b5599d0e94b2dc06: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:35.080698 20215 raft_consensus.cc:729] T b315327141954b53a0f87c2d1dc14b9b P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Initialized, Role: FOLLOWER
I20240502 14:26:35.081629 20214 tablet_bootstrap.cc:492] T b315327141954b53a0f87c2d1dc14b9b P 007b2e79dc424a6f80141180f072f2d8: Bootstrap starting.
I20240502 14:26:35.082605 20218 tablet_bootstrap.cc:654] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:35.082764 20216 tablet_bootstrap.cc:492] T 8a2bed29fcd64f7bbb04286404ef09de P c902805060df4df9b5599d0e94b2dc06: No bootstrap required, opened a new log
I20240502 14:26:35.083003 20216 ts_tablet_manager.cc:1367] T 8a2bed29fcd64f7bbb04286404ef09de P c902805060df4df9b5599d0e94b2dc06: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 14:26:35.083824 20215 consensus_queue.cc:260] T b315327141954b53a0f87c2d1dc14b9b P fec04c50dc5b49878ffa3c3873df32e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:35.083937 20216 raft_consensus.cc:348] T 8a2bed29fcd64f7bbb04286404ef09de P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:35.084306 20216 raft_consensus.cc:374] T 8a2bed29fcd64f7bbb04286404ef09de P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:35.084427 20216 raft_consensus.cc:729] T 8a2bed29fcd64f7bbb04286404ef09de P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Initialized, Role: FOLLOWER
I20240502 14:26:35.084699 20216 consensus_queue.cc:260] T 8a2bed29fcd64f7bbb04286404ef09de P c902805060df4df9b5599d0e94b2dc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:35.084853 20214 tablet_bootstrap.cc:654] T b315327141954b53a0f87c2d1dc14b9b P 007b2e79dc424a6f80141180f072f2d8: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:35.085067 20215 ts_tablet_manager.cc:1398] T b315327141954b53a0f87c2d1dc14b9b P fec04c50dc5b49878ffa3c3873df32e4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:26:35.085381 20215 tablet_bootstrap.cc:492] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4: Bootstrap starting.
I20240502 14:26:35.086217 20216 ts_tablet_manager.cc:1398] T 8a2bed29fcd64f7bbb04286404ef09de P c902805060df4df9b5599d0e94b2dc06: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:26:35.089356 20215 tablet_bootstrap.cc:654] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:35.091548 20218 tablet_bootstrap.cc:492] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b: No bootstrap required, opened a new log
I20240502 14:26:35.091686 20218 ts_tablet_manager.cc:1367] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240502 14:26:35.092350 20218 raft_consensus.cc:348] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:35.092563 20218 raft_consensus.cc:374] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:35.092679 20218 raft_consensus.cc:729] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Initialized, Role: FOLLOWER
I20240502 14:26:35.092727 20215 tablet_bootstrap.cc:492] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4: No bootstrap required, opened a new log
I20240502 14:26:35.092916 20215 ts_tablet_manager.cc:1367] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.002s
I20240502 14:26:35.092980 20218 consensus_queue.cc:260] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:35.093149 20214 tablet_bootstrap.cc:492] T b315327141954b53a0f87c2d1dc14b9b P 007b2e79dc424a6f80141180f072f2d8: No bootstrap required, opened a new log
I20240502 14:26:35.093335 20214 ts_tablet_manager.cc:1367] T b315327141954b53a0f87c2d1dc14b9b P 007b2e79dc424a6f80141180f072f2d8: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20240502 14:26:35.093838 20215 raft_consensus.cc:348] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:35.094075 20215 raft_consensus.cc:374] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:35.094162 20215 raft_consensus.cc:729] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Initialized, Role: FOLLOWER
I20240502 14:26:35.094175 20214 raft_consensus.cc:348] T b315327141954b53a0f87c2d1dc14b9b P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:35.094360 20214 raft_consensus.cc:374] T b315327141954b53a0f87c2d1dc14b9b P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:35.094445 20214 raft_consensus.cc:729] T b315327141954b53a0f87c2d1dc14b9b P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Initialized, Role: FOLLOWER
I20240502 14:26:35.094626 20215 consensus_queue.cc:260] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:35.094689 20214 consensus_queue.cc:260] T b315327141954b53a0f87c2d1dc14b9b P 007b2e79dc424a6f80141180f072f2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:35.095212 20214 ts_tablet_manager.cc:1398] T b315327141954b53a0f87c2d1dc14b9b P 007b2e79dc424a6f80141180f072f2d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:26:35.095252 20215 ts_tablet_manager.cc:1398] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:26:35.095535 20215 tablet_bootstrap.cc:492] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4: Bootstrap starting.
I20240502 14:26:35.096153 20214 tablet_bootstrap.cc:492] T 8a2bed29fcd64f7bbb04286404ef09de P 007b2e79dc424a6f80141180f072f2d8: Bootstrap starting.
I20240502 14:26:35.097389 20215 tablet_bootstrap.cc:654] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:35.099128 20218 ts_tablet_manager.cc:1398] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 14:26:35.099393 20214 tablet_bootstrap.cc:654] T 8a2bed29fcd64f7bbb04286404ef09de P 007b2e79dc424a6f80141180f072f2d8: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:35.101727 20203 raft_consensus.cc:482] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:35.101917 20203 raft_consensus.cc:504] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:35.102674 20203 leader_election.cc:288] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463), c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:26:35.103735 19097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2bed29fcd64f7bbb04286404ef09de" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b2e79dc424a6f80141180f072f2d8" is_pre_election: true
I20240502 14:26:35.104338 20218 tablet_bootstrap.cc:492] T e5fb3b049ba94af998791640bd0e31c5 P 2db304b538e14857b7b1cc272d0db94b: Bootstrap starting.
I20240502 14:26:35.104485 18964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2bed29fcd64f7bbb04286404ef09de" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06" is_pre_election: true
I20240502 14:26:35.104748 18964 raft_consensus.cc:2454] T 8a2bed29fcd64f7bbb04286404ef09de P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 0.
I20240502 14:26:35.105310 19921 raft_consensus.cc:482] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:35.105468 19921 raft_consensus.cc:504] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
W20240502 14:26:35.105499 18769 leader_election.cc:341] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:26:35.105922 18769 leader_election.cc:302] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c902805060df4df9b5599d0e94b2dc06, fec04c50dc5b49878ffa3c3873df32e4; no voters: 007b2e79dc424a6f80141180f072f2d8
I20240502 14:26:35.106195 19921 leader_election.cc:288] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009), 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:26:35.106385 20203 raft_consensus.cc:2781] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:35.106501 20203 raft_consensus.cc:482] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:35.106600 20203 raft_consensus.cc:3037] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:35.106817 18832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b315327141954b53a0f87c2d1dc14b9b" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec04c50dc5b49878ffa3c3873df32e4" is_pre_election: true
I20240502 14:26:35.107033 18832 raft_consensus.cc:2454] T b315327141954b53a0f87c2d1dc14b9b P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2db304b538e14857b7b1cc272d0db94b in term 0.
I20240502 14:26:35.107950 18634 leader_election.cc:302] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b, fec04c50dc5b49878ffa3c3873df32e4; no voters:
I20240502 14:26:35.108597 19097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b315327141954b53a0f87c2d1dc14b9b" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b2e79dc424a6f80141180f072f2d8" is_pre_election: true
I20240502 14:26:35.108815 19097 raft_consensus.cc:2454] T b315327141954b53a0f87c2d1dc14b9b P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2db304b538e14857b7b1cc272d0db94b in term 0.
I20240502 14:26:35.109565 19921 raft_consensus.cc:2781] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:35.109665 19921 raft_consensus.cc:482] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:35.109546 20203 raft_consensus.cc:504] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:35.109738 19921 raft_consensus.cc:3037] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:35.111646 19097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2bed29fcd64f7bbb04286404ef09de" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b2e79dc424a6f80141180f072f2d8"
W20240502 14:26:35.112197 18769 leader_election.cc:341] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:26:35.112404 19921 raft_consensus.cc:504] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:35.113651 18832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b315327141954b53a0f87c2d1dc14b9b" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec04c50dc5b49878ffa3c3873df32e4"
I20240502 14:26:35.113874 18832 raft_consensus.cc:3037] T b315327141954b53a0f87c2d1dc14b9b P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:35.113934 18964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2bed29fcd64f7bbb04286404ef09de" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06"
I20240502 14:26:35.114192 18964 raft_consensus.cc:3037] T 8a2bed29fcd64f7bbb04286404ef09de P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:35.116839 18832 raft_consensus.cc:2454] T b315327141954b53a0f87c2d1dc14b9b P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2db304b538e14857b7b1cc272d0db94b in term 1.
I20240502 14:26:35.116886 18964 raft_consensus.cc:2454] T 8a2bed29fcd64f7bbb04286404ef09de P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 1.
I20240502 14:26:35.117440 18769 leader_election.cc:302] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c902805060df4df9b5599d0e94b2dc06, fec04c50dc5b49878ffa3c3873df32e4; no voters: 007b2e79dc424a6f80141180f072f2d8
I20240502 14:26:35.117810 20097 raft_consensus.cc:2781] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:26:35.117964 20097 raft_consensus.cc:686] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Becoming Leader. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Running, Role: LEADER
I20240502 14:26:35.118330 20097 consensus_queue.cc:237] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:35.119374 19921 leader_election.cc:288] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 election: Requested vote from peers fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009), 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:26:35.120525 19097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b315327141954b53a0f87c2d1dc14b9b" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b2e79dc424a6f80141180f072f2d8"
I20240502 14:26:35.120841 19097 raft_consensus.cc:3037] T b315327141954b53a0f87c2d1dc14b9b P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:35.120859 20203 leader_election.cc:288] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 election: Requested vote from peers 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463), c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:26:35.124086 19097 raft_consensus.cc:2454] T b315327141954b53a0f87c2d1dc14b9b P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2db304b538e14857b7b1cc272d0db94b in term 1.
I20240502 14:26:35.124603 18637 leader_election.cc:302] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 007b2e79dc424a6f80141180f072f2d8, 2db304b538e14857b7b1cc272d0db94b; no voters:
I20240502 14:26:35.125039 19921 raft_consensus.cc:2781] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:26:35.125244 19921 raft_consensus.cc:686] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Becoming Leader. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Running, Role: LEADER
I20240502 14:26:35.125597 19921 consensus_queue.cc:237] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:35.132859 20218 tablet_bootstrap.cc:654] T e5fb3b049ba94af998791640bd0e31c5 P 2db304b538e14857b7b1cc272d0db94b: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:35.137373 18560 catalog_manager.cc:5484] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66). New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
I20240502 14:26:35.144922 20218 tablet_bootstrap.cc:492] T e5fb3b049ba94af998791640bd0e31c5 P 2db304b538e14857b7b1cc272d0db94b: No bootstrap required, opened a new log
I20240502 14:26:35.145171 20218 ts_tablet_manager.cc:1367] T e5fb3b049ba94af998791640bd0e31c5 P 2db304b538e14857b7b1cc272d0db94b: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.002s
I20240502 14:26:35.146051 20218 raft_consensus.cc:348] T e5fb3b049ba94af998791640bd0e31c5 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:35.146348 20218 raft_consensus.cc:374] T e5fb3b049ba94af998791640bd0e31c5 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:35.146469 20218 raft_consensus.cc:729] T e5fb3b049ba94af998791640bd0e31c5 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Initialized, Role: FOLLOWER
I20240502 14:26:35.146775 20218 consensus_queue.cc:260] T e5fb3b049ba94af998791640bd0e31c5 P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:35.146706 18560 catalog_manager.cc:5484] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2db304b538e14857b7b1cc272d0db94b (127.5.107.65). New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:26:35.147387 20218 ts_tablet_manager.cc:1398] T e5fb3b049ba94af998791640bd0e31c5 P 2db304b538e14857b7b1cc272d0db94b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:26:35.149534 20215 tablet_bootstrap.cc:492] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4: No bootstrap required, opened a new log
I20240502 14:26:35.149705 20215 ts_tablet_manager.cc:1367] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4: Time spent bootstrapping tablet: real 0.054s user 0.021s sys 0.032s
I20240502 14:26:35.150357 20214 tablet_bootstrap.cc:492] T 8a2bed29fcd64f7bbb04286404ef09de P 007b2e79dc424a6f80141180f072f2d8: No bootstrap required, opened a new log
I20240502 14:26:35.150586 20214 ts_tablet_manager.cc:1367] T 8a2bed29fcd64f7bbb04286404ef09de P 007b2e79dc424a6f80141180f072f2d8: Time spent bootstrapping tablet: real 0.054s user 0.017s sys 0.032s
I20240502 14:26:35.150578 20215 raft_consensus.cc:348] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:35.150801 20215 raft_consensus.cc:374] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:35.150888 20215 raft_consensus.cc:729] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Initialized, Role: FOLLOWER
I20240502 14:26:35.151118 20215 consensus_queue.cc:260] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:35.151815 20215 ts_tablet_manager.cc:1398] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:26:35.151762 20214 raft_consensus.cc:348] T 8a2bed29fcd64f7bbb04286404ef09de P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:35.152024 20214 raft_consensus.cc:374] T 8a2bed29fcd64f7bbb04286404ef09de P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:35.152107 20214 raft_consensus.cc:729] T 8a2bed29fcd64f7bbb04286404ef09de P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Initialized, Role: FOLLOWER
I20240502 14:26:35.152355 20214 consensus_queue.cc:260] T 8a2bed29fcd64f7bbb04286404ef09de P 007b2e79dc424a6f80141180f072f2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:35.153488 20214 ts_tablet_manager.cc:1398] T 8a2bed29fcd64f7bbb04286404ef09de P 007b2e79dc424a6f80141180f072f2d8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:26:35.153798 20214 tablet_bootstrap.cc:492] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8: Bootstrap starting.
I20240502 14:26:35.155354 20214 tablet_bootstrap.cc:654] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:35.157130 20214 tablet_bootstrap.cc:492] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8: No bootstrap required, opened a new log
I20240502 14:26:35.157291 20214 ts_tablet_manager.cc:1367] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20240502 14:26:35.157900 20214 raft_consensus.cc:348] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:35.158049 20214 raft_consensus.cc:374] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:35.158111 20214 raft_consensus.cc:729] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Initialized, Role: FOLLOWER
I20240502 14:26:35.158282 20214 consensus_queue.cc:260] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:35.158736 20214 ts_tablet_manager.cc:1398] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:26:35.244710 20180 raft_consensus.cc:482] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:35.244926 20180 raft_consensus.cc:504] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:35.245728 20180 leader_election.cc:288] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009), 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:26:35.246479 18700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5fb3b049ba94af998791640bd0e31c5" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b" is_pre_election: true
I20240502 14:26:35.246480 18832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5fb3b049ba94af998791640bd0e31c5" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec04c50dc5b49878ffa3c3873df32e4" is_pre_election: true
I20240502 14:26:35.246788 18832 raft_consensus.cc:2454] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 0.
I20240502 14:26:35.246794 18700 raft_consensus.cc:2454] T e5fb3b049ba94af998791640bd0e31c5 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 0.
I20240502 14:26:35.247323 19029 leader_election.cc:302] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 007b2e79dc424a6f80141180f072f2d8, 2db304b538e14857b7b1cc272d0db94b, fec04c50dc5b49878ffa3c3873df32e4; no voters:
I20240502 14:26:35.247727 20180 raft_consensus.cc:2781] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:35.247826 20180 raft_consensus.cc:482] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:35.247886 20180 raft_consensus.cc:3037] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:35.249739 20180 raft_consensus.cc:504] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:35.250260 20180 leader_election.cc:288] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 election: Requested vote from peers fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009), 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:26:35.250931 18700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5fb3b049ba94af998791640bd0e31c5" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b"
I20240502 14:26:35.250931 18832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5fb3b049ba94af998791640bd0e31c5" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec04c50dc5b49878ffa3c3873df32e4"
I20240502 14:26:35.251178 18700 raft_consensus.cc:3037] T e5fb3b049ba94af998791640bd0e31c5 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:35.251166 18832 raft_consensus.cc:3037] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:35.254215 18832 raft_consensus.cc:2454] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 1.
I20240502 14:26:35.254215 18700 raft_consensus.cc:2454] T e5fb3b049ba94af998791640bd0e31c5 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 1.
I20240502 14:26:35.254724 19029 leader_election.cc:302] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 007b2e79dc424a6f80141180f072f2d8, fec04c50dc5b49878ffa3c3873df32e4; no voters:
I20240502 14:26:35.255278 20180 raft_consensus.cc:2781] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:26:35.255441 20180 raft_consensus.cc:686] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Becoming Leader. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Running, Role: LEADER
I20240502 14:26:35.255707 20180 consensus_queue.cc:237] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:35.258965 18560 catalog_manager.cc:5484] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68). New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:26:35.274645 20180 raft_consensus.cc:482] T 8a2bed29fcd64f7bbb04286404ef09de P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:35.274850 20180 raft_consensus.cc:504] T 8a2bed29fcd64f7bbb04286404ef09de P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:35.275460 20180 leader_election.cc:288] T 8a2bed29fcd64f7bbb04286404ef09de P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009), c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:26:35.275955 18832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2bed29fcd64f7bbb04286404ef09de" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec04c50dc5b49878ffa3c3873df32e4" is_pre_election: true
I20240502 14:26:35.276008 18964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2bed29fcd64f7bbb04286404ef09de" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06" is_pre_election: true
I20240502 14:26:35.276271 18964 raft_consensus.cc:2379] T 8a2bed29fcd64f7bbb04286404ef09de P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 007b2e79dc424a6f80141180f072f2d8 in current term 1: Already voted for candidate fec04c50dc5b49878ffa3c3873df32e4 in this term.
I20240502 14:26:35.276768 19032 leader_election.cc:302] T 8a2bed29fcd64f7bbb04286404ef09de P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 007b2e79dc424a6f80141180f072f2d8; no voters: c902805060df4df9b5599d0e94b2dc06, fec04c50dc5b49878ffa3c3873df32e4
I20240502 14:26:35.277057 20180 raft_consensus.cc:3037] T 8a2bed29fcd64f7bbb04286404ef09de P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:35.279269 20180 raft_consensus.cc:2726] T 8a2bed29fcd64f7bbb04286404ef09de P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 14:26:35.650027 20231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:26:35.650410 20231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 14:26:35.656615 19921 consensus_queue.cc:1035] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:26:35.658388 20231 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:26:35.662992 19921 consensus_queue.cc:1035] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:35.670720 20241 consensus_queue.cc:1035] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:35.671196 20097 consensus_queue.cc:1035] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:35.703919 20180 consensus_queue.cc:1035] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:35.704588 20088 consensus_queue.cc:1035] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'
INSERT report
rows total: 2000
time total: 277.418 ms
time per row: 0.138709 ms
I20240502 14:26:36.112838 18552 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48228:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:26:36.122604 18681 tablet_service.cc:1467] Processing CreateTablet for tablet 8b8d81c275144f898dcc69a6d98d11e0 (DEFAULT_TABLE table=rebalance_test_table_7 [id=b260896c8b8744c0a1b4f691850aec93]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:36.123085 18681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b8d81c275144f898dcc69a6d98d11e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:36.123389 18813 tablet_service.cc:1467] Processing CreateTablet for tablet 8b8d81c275144f898dcc69a6d98d11e0 (DEFAULT_TABLE table=rebalance_test_table_7 [id=b260896c8b8744c0a1b4f691850aec93]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:36.123862 18813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b8d81c275144f898dcc69a6d98d11e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:36.126008 18812 tablet_service.cc:1467] Processing CreateTablet for tablet 5be0d6f4f1874c79b906c4bdc34b0ce9 (DEFAULT_TABLE table=rebalance_test_table_7 [id=b260896c8b8744c0a1b4f691850aec93]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:36.126380 18812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be0d6f4f1874c79b906c4bdc34b0ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:36.126645 18680 tablet_service.cc:1467] Processing CreateTablet for tablet df5f14206a1d46e4866f34fa234f945b (DEFAULT_TABLE table=rebalance_test_table_7 [id=b260896c8b8744c0a1b4f691850aec93]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:36.126996 18680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df5f14206a1d46e4866f34fa234f945b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:36.129945 19077 tablet_service.cc:1467] Processing CreateTablet for tablet 5be0d6f4f1874c79b906c4bdc34b0ce9 (DEFAULT_TABLE table=rebalance_test_table_7 [id=b260896c8b8744c0a1b4f691850aec93]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:36.130057 18944 tablet_service.cc:1467] Processing CreateTablet for tablet 5be0d6f4f1874c79b906c4bdc34b0ce9 (DEFAULT_TABLE table=rebalance_test_table_7 [id=b260896c8b8744c0a1b4f691850aec93]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:36.130196 18943 tablet_service.cc:1467] Processing CreateTablet for tablet 8b8d81c275144f898dcc69a6d98d11e0 (DEFAULT_TABLE table=rebalance_test_table_7 [id=b260896c8b8744c0a1b4f691850aec93]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:36.130826 18945 tablet_service.cc:1467] Processing CreateTablet for tablet df5f14206a1d46e4866f34fa234f945b (DEFAULT_TABLE table=rebalance_test_table_7 [id=b260896c8b8744c0a1b4f691850aec93]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:36.131059 19077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be0d6f4f1874c79b906c4bdc34b0ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:36.130478 18944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be0d6f4f1874c79b906c4bdc34b0ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:36.131291 19076 tablet_service.cc:1467] Processing CreateTablet for tablet df5f14206a1d46e4866f34fa234f945b (DEFAULT_TABLE table=rebalance_test_table_7 [id=b260896c8b8744c0a1b4f691850aec93]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:36.131662 19076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df5f14206a1d46e4866f34fa234f945b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:36.133419 18945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df5f14206a1d46e4866f34fa234f945b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:36.134478 18943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b8d81c275144f898dcc69a6d98d11e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:36.140830 20295 tablet_bootstrap.cc:492] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06: Bootstrap starting.
I20240502 14:26:36.142915 20295 tablet_bootstrap.cc:654] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:36.144371 20296 tablet_bootstrap.cc:492] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b: Bootstrap starting.
I20240502 14:26:36.147579 20296 tablet_bootstrap.cc:654] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:36.150601 20294 tablet_bootstrap.cc:492] T df5f14206a1d46e4866f34fa234f945b P 007b2e79dc424a6f80141180f072f2d8: Bootstrap starting.
I20240502 14:26:36.152271 20293 tablet_bootstrap.cc:492] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4: Bootstrap starting.
I20240502 14:26:36.152954 20294 tablet_bootstrap.cc:654] T df5f14206a1d46e4866f34fa234f945b P 007b2e79dc424a6f80141180f072f2d8: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:36.155022 20293 tablet_bootstrap.cc:654] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:36.155472 20295 tablet_bootstrap.cc:492] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06: No bootstrap required, opened a new log
I20240502 14:26:36.155628 20295 ts_tablet_manager.cc:1367] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20240502 14:26:36.156035 20294 tablet_bootstrap.cc:492] T df5f14206a1d46e4866f34fa234f945b P 007b2e79dc424a6f80141180f072f2d8: No bootstrap required, opened a new log
I20240502 14:26:36.156212 20294 ts_tablet_manager.cc:1367] T df5f14206a1d46e4866f34fa234f945b P 007b2e79dc424a6f80141180f072f2d8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 14:26:36.156502 20295 raft_consensus.cc:348] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:36.156778 20295 raft_consensus.cc:374] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:36.156905 20295 raft_consensus.cc:729] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Initialized, Role: FOLLOWER
I20240502 14:26:36.157054 20294 raft_consensus.cc:348] T df5f14206a1d46e4866f34fa234f945b P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:36.157295 20294 raft_consensus.cc:374] T df5f14206a1d46e4866f34fa234f945b P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:36.157413 20294 raft_consensus.cc:729] T df5f14206a1d46e4866f34fa234f945b P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Initialized, Role: FOLLOWER
I20240502 14:26:36.157296 20295 consensus_queue.cc:260] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:36.157722 20294 consensus_queue.cc:260] T df5f14206a1d46e4866f34fa234f945b P 007b2e79dc424a6f80141180f072f2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:36.158391 20294 ts_tablet_manager.cc:1398] T df5f14206a1d46e4866f34fa234f945b P 007b2e79dc424a6f80141180f072f2d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:26:36.158763 20295 ts_tablet_manager.cc:1398] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:26:36.159305 20295 tablet_bootstrap.cc:492] T 8b8d81c275144f898dcc69a6d98d11e0 P c902805060df4df9b5599d0e94b2dc06: Bootstrap starting.
I20240502 14:26:36.159897 20293 tablet_bootstrap.cc:492] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4: No bootstrap required, opened a new log
I20240502 14:26:36.160095 20293 ts_tablet_manager.cc:1367] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20240502 14:26:36.160562 20294 tablet_bootstrap.cc:492] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P 007b2e79dc424a6f80141180f072f2d8: Bootstrap starting.
I20240502 14:26:36.160967 20293 raft_consensus.cc:348] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:36.161247 20293 raft_consensus.cc:374] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:36.161370 20293 raft_consensus.cc:729] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Initialized, Role: FOLLOWER
I20240502 14:26:36.161685 20293 consensus_queue.cc:260] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:36.162241 20294 tablet_bootstrap.cc:654] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P 007b2e79dc424a6f80141180f072f2d8: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:36.163106 20296 tablet_bootstrap.cc:492] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b: No bootstrap required, opened a new log
I20240502 14:26:36.163323 20296 ts_tablet_manager.cc:1367] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.003s
I20240502 14:26:36.163899 20293 ts_tablet_manager.cc:1398] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 14:26:36.164225 20293 tablet_bootstrap.cc:492] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4: Bootstrap starting.
I20240502 14:26:36.164502 20295 tablet_bootstrap.cc:654] T 8b8d81c275144f898dcc69a6d98d11e0 P c902805060df4df9b5599d0e94b2dc06: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:36.165613 20296 raft_consensus.cc:348] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:36.165879 20296 raft_consensus.cc:374] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:36.165993 20296 raft_consensus.cc:729] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Initialized, Role: FOLLOWER
I20240502 14:26:36.166039 20293 tablet_bootstrap.cc:654] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:36.166263 20296 consensus_queue.cc:260] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:36.167078 20296 ts_tablet_manager.cc:1398] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:26:36.167395 20296 tablet_bootstrap.cc:492] T 8b8d81c275144f898dcc69a6d98d11e0 P 2db304b538e14857b7b1cc272d0db94b: Bootstrap starting.
I20240502 14:26:36.168977 20296 tablet_bootstrap.cc:654] T 8b8d81c275144f898dcc69a6d98d11e0 P 2db304b538e14857b7b1cc272d0db94b: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:36.176446 20295 tablet_bootstrap.cc:492] T 8b8d81c275144f898dcc69a6d98d11e0 P c902805060df4df9b5599d0e94b2dc06: No bootstrap required, opened a new log
I20240502 14:26:36.176782 20295 ts_tablet_manager.cc:1367] T 8b8d81c275144f898dcc69a6d98d11e0 P c902805060df4df9b5599d0e94b2dc06: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.004s
I20240502 14:26:36.177731 20296 tablet_bootstrap.cc:492] T 8b8d81c275144f898dcc69a6d98d11e0 P 2db304b538e14857b7b1cc272d0db94b: No bootstrap required, opened a new log
I20240502 14:26:36.177919 20296 ts_tablet_manager.cc:1367] T 8b8d81c275144f898dcc69a6d98d11e0 P 2db304b538e14857b7b1cc272d0db94b: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20240502 14:26:36.177852 20295 raft_consensus.cc:348] T 8b8d81c275144f898dcc69a6d98d11e0 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:36.178206 20295 raft_consensus.cc:374] T 8b8d81c275144f898dcc69a6d98d11e0 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:36.178328 20295 raft_consensus.cc:729] T 8b8d81c275144f898dcc69a6d98d11e0 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Initialized, Role: FOLLOWER
I20240502 14:26:36.178596 20294 tablet_bootstrap.cc:492] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P 007b2e79dc424a6f80141180f072f2d8: No bootstrap required, opened a new log
I20240502 14:26:36.178786 20293 tablet_bootstrap.cc:492] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4: No bootstrap required, opened a new log
I20240502 14:26:36.178669 20295 consensus_queue.cc:260] T 8b8d81c275144f898dcc69a6d98d11e0 P c902805060df4df9b5599d0e94b2dc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:36.178810 20296 raft_consensus.cc:348] T 8b8d81c275144f898dcc69a6d98d11e0 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:36.178943 20294 ts_tablet_manager.cc:1367] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P 007b2e79dc424a6f80141180f072f2d8: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.007s
I20240502 14:26:36.179018 20293 ts_tablet_manager.cc:1367] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20240502 14:26:36.179085 20296 raft_consensus.cc:374] T 8b8d81c275144f898dcc69a6d98d11e0 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:36.179234 20296 raft_consensus.cc:729] T 8b8d81c275144f898dcc69a6d98d11e0 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Initialized, Role: FOLLOWER
I20240502 14:26:36.179520 20296 consensus_queue.cc:260] T 8b8d81c275144f898dcc69a6d98d11e0 P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:36.179750 20294 raft_consensus.cc:348] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:36.179986 20294 raft_consensus.cc:374] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:36.180073 20294 raft_consensus.cc:729] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Initialized, Role: FOLLOWER
I20240502 14:26:36.180208 20295 ts_tablet_manager.cc:1398] T 8b8d81c275144f898dcc69a6d98d11e0 P c902805060df4df9b5599d0e94b2dc06: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:26:36.180352 20294 consensus_queue.cc:260] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P 007b2e79dc424a6f80141180f072f2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:36.180558 20295 tablet_bootstrap.cc:492] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06: Bootstrap starting.
I20240502 14:26:36.180974 20296 ts_tablet_manager.cc:1398] T 8b8d81c275144f898dcc69a6d98d11e0 P 2db304b538e14857b7b1cc272d0db94b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:26:36.182283 20294 ts_tablet_manager.cc:1398] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P 007b2e79dc424a6f80141180f072f2d8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:26:36.182261 20293 raft_consensus.cc:348] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:36.182551 20293 raft_consensus.cc:374] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:36.182665 20293 raft_consensus.cc:729] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Initialized, Role: FOLLOWER
I20240502 14:26:36.182932 20293 consensus_queue.cc:260] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:36.183190 20295 tablet_bootstrap.cc:654] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:36.183539 20293 ts_tablet_manager.cc:1398] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:26:36.189344 20295 tablet_bootstrap.cc:492] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06: No bootstrap required, opened a new log
I20240502 14:26:36.189520 20295 ts_tablet_manager.cc:1367] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20240502 14:26:36.190294 20295 raft_consensus.cc:348] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:36.190532 20295 raft_consensus.cc:374] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:36.190627 20295 raft_consensus.cc:729] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Initialized, Role: FOLLOWER
I20240502 14:26:36.190868 20295 consensus_queue.cc:260] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:36.191438 20295 ts_tablet_manager.cc:1398] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:26:36.247866 20236 raft_consensus.cc:482] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:36.248126 20236 raft_consensus.cc:504] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:36.248832 20236 leader_election.cc:288] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409), 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:26:36.249336 18964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df5f14206a1d46e4866f34fa234f945b" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06" is_pre_election: true
I20240502 14:26:36.249444 19096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df5f14206a1d46e4866f34fa234f945b" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b2e79dc424a6f80141180f072f2d8" is_pre_election: true
I20240502 14:26:36.249593 18964 raft_consensus.cc:2454] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2db304b538e14857b7b1cc272d0db94b in term 0.
I20240502 14:26:36.249670 19096 raft_consensus.cc:2454] T df5f14206a1d46e4866f34fa234f945b P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2db304b538e14857b7b1cc272d0db94b in term 0.
I20240502 14:26:36.250059 18637 leader_election.cc:302] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 007b2e79dc424a6f80141180f072f2d8, 2db304b538e14857b7b1cc272d0db94b; no voters:
I20240502 14:26:36.250298 20236 raft_consensus.cc:2781] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:36.250417 20236 raft_consensus.cc:482] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:36.250504 20236 raft_consensus.cc:3037] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:36.252415 20236 raft_consensus.cc:504] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:36.252983 20236 leader_election.cc:288] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 election: Requested vote from peers c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409), 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:26:36.253518 18964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df5f14206a1d46e4866f34fa234f945b" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06"
I20240502 14:26:36.253567 19096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df5f14206a1d46e4866f34fa234f945b" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b2e79dc424a6f80141180f072f2d8"
I20240502 14:26:36.253732 18964 raft_consensus.cc:3037] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:36.253763 19096 raft_consensus.cc:3037] T df5f14206a1d46e4866f34fa234f945b P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:36.255899 18964 raft_consensus.cc:2454] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2db304b538e14857b7b1cc272d0db94b in term 1.
I20240502 14:26:36.255998 19096 raft_consensus.cc:2454] T df5f14206a1d46e4866f34fa234f945b P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2db304b538e14857b7b1cc272d0db94b in term 1.
I20240502 14:26:36.256333 18637 leader_election.cc:302] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b, c902805060df4df9b5599d0e94b2dc06; no voters:
I20240502 14:26:36.256639 20236 raft_consensus.cc:2781] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:26:36.256764 20236 raft_consensus.cc:686] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Becoming Leader. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Running, Role: LEADER
I20240502 14:26:36.257081 20236 consensus_queue.cc:237] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:36.259970 18553 catalog_manager.cc:5484] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2db304b538e14857b7b1cc272d0db94b (127.5.107.65). New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:26:36.457901 20097 raft_consensus.cc:482] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:36.458106 20097 raft_consensus.cc:504] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:36.458770 20097 leader_election.cc:288] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749), c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:26:36.459291 18964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b8d81c275144f898dcc69a6d98d11e0" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06" is_pre_election: true
I20240502 14:26:36.459311 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b8d81c275144f898dcc69a6d98d11e0" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b" is_pre_election: true
I20240502 14:26:36.459563 18701 raft_consensus.cc:2454] T 8b8d81c275144f898dcc69a6d98d11e0 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 0.
I20240502 14:26:36.459547 18964 raft_consensus.cc:2454] T 8b8d81c275144f898dcc69a6d98d11e0 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 0.
I20240502 14:26:36.459990 18769 leader_election.cc:302] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c902805060df4df9b5599d0e94b2dc06, fec04c50dc5b49878ffa3c3873df32e4; no voters:
I20240502 14:26:36.460300 20097 raft_consensus.cc:2781] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:36.460430 20097 raft_consensus.cc:482] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:36.460505 20097 raft_consensus.cc:3037] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:36.462401 20097 raft_consensus.cc:504] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:36.462947 20097 leader_election.cc:288] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 election: Requested vote from peers 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749), c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:26:36.463428 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b8d81c275144f898dcc69a6d98d11e0" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b"
I20240502 14:26:36.463483 18964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b8d81c275144f898dcc69a6d98d11e0" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06"
I20240502 14:26:36.463634 18701 raft_consensus.cc:3037] T 8b8d81c275144f898dcc69a6d98d11e0 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:36.463696 18964 raft_consensus.cc:3037] T 8b8d81c275144f898dcc69a6d98d11e0 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:36.465638 18964 raft_consensus.cc:2454] T 8b8d81c275144f898dcc69a6d98d11e0 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 1.
I20240502 14:26:36.465684 18701 raft_consensus.cc:2454] T 8b8d81c275144f898dcc69a6d98d11e0 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 1.
I20240502 14:26:36.466050 18766 leader_election.cc:302] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b, fec04c50dc5b49878ffa3c3873df32e4; no voters:
I20240502 14:26:36.466351 20097 raft_consensus.cc:2781] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:26:36.467095 20097 raft_consensus.cc:686] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Becoming Leader. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Running, Role: LEADER
I20240502 14:26:36.467474 20097 consensus_queue.cc:237] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:36.475171 18553 catalog_manager.cc:5484] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66). New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
I20240502 14:26:36.531570 20308 raft_consensus.cc:482] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:36.531819 20308 raft_consensus.cc:504] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:36.532505 20308 leader_election.cc:288] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409), 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:26:36.533079 19096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be0d6f4f1874c79b906c4bdc34b0ce9" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b2e79dc424a6f80141180f072f2d8" is_pre_election: true
I20240502 14:26:36.533085 18964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be0d6f4f1874c79b906c4bdc34b0ce9" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06" is_pre_election: true
I20240502 14:26:36.533366 19096 raft_consensus.cc:2454] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 0.
I20240502 14:26:36.533372 18964 raft_consensus.cc:2454] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 0.
I20240502 14:26:36.533771 18769 leader_election.cc:302] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c902805060df4df9b5599d0e94b2dc06, fec04c50dc5b49878ffa3c3873df32e4; no voters:
I20240502 14:26:36.534086 20308 raft_consensus.cc:2781] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:36.534215 20308 raft_consensus.cc:482] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:36.534337 20308 raft_consensus.cc:3037] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:36.536302 20308 raft_consensus.cc:504] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:36.536872 20308 leader_election.cc:288] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 election: Requested vote from peers c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409), 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:26:36.537498 19096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be0d6f4f1874c79b906c4bdc34b0ce9" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b2e79dc424a6f80141180f072f2d8"
I20240502 14:26:36.537528 18964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be0d6f4f1874c79b906c4bdc34b0ce9" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06"
I20240502 14:26:36.537734 19096 raft_consensus.cc:3037] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:36.537787 18964 raft_consensus.cc:3037] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:36.539726 19096 raft_consensus.cc:2454] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 1.
I20240502 14:26:36.539739 18964 raft_consensus.cc:2454] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 1.
I20240502 14:26:36.540180 18769 leader_election.cc:302] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c902805060df4df9b5599d0e94b2dc06, fec04c50dc5b49878ffa3c3873df32e4; no voters:
I20240502 14:26:36.540571 20308 raft_consensus.cc:2781] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:26:36.540740 20308 raft_consensus.cc:686] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Becoming Leader. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Running, Role: LEADER
I20240502 14:26:36.541057 20308 consensus_queue.cc:237] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:36.544379 18553 catalog_manager.cc:5484] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66). New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:26:36.668166 20236 consensus_queue.cc:1035] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:36.674759 20236 consensus_queue.cc:1035] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:36.884959 20308 consensus_queue.cc:1035] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:36.890995 20097 consensus_queue.cc:1035] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:36.925586 20308 consensus_queue.cc:1035] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:36.934273 20308 consensus_queue.cc:1035] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:26:36.986116 20313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:26:36.986485 20313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:26:36.994066 20313 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
W20240502 14:26:37.237520 18766 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.65:36749, user_credentials={real_user=slave}} blocked reactor thread for 65315.1us
INSERT report
rows total: 2000
time total: 259.01 ms
time per row: 0.129505 ms
I20240502 14:26:37.400965 18558 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48228:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:26:37.410215 18680 tablet_service.cc:1467] Processing CreateTablet for tablet 51ef0be93c914e258635e13f3cca79aa (DEFAULT_TABLE table=rebalance_test_table_8 [id=5a8a0451cdf0479c94edf1d94e77fb83]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:37.410566 18944 tablet_service.cc:1467] Processing CreateTablet for tablet 51ef0be93c914e258635e13f3cca79aa (DEFAULT_TABLE table=rebalance_test_table_8 [id=5a8a0451cdf0479c94edf1d94e77fb83]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:37.410753 18680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51ef0be93c914e258635e13f3cca79aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:37.410804 18812 tablet_service.cc:1467] Processing CreateTablet for tablet db56136f2dbb429098d4d78b15a3a439 (DEFAULT_TABLE table=rebalance_test_table_8 [id=5a8a0451cdf0479c94edf1d94e77fb83]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:37.411002 18944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51ef0be93c914e258635e13f3cca79aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:37.411186 18812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db56136f2dbb429098d4d78b15a3a439. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:37.411401 18945 tablet_service.cc:1467] Processing CreateTablet for tablet 21af917743444f2db932636a78f33512 (DEFAULT_TABLE table=rebalance_test_table_8 [id=5a8a0451cdf0479c94edf1d94e77fb83]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:37.411785 18945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21af917743444f2db932636a78f33512. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:37.413610 18813 tablet_service.cc:1467] Processing CreateTablet for tablet 51ef0be93c914e258635e13f3cca79aa (DEFAULT_TABLE table=rebalance_test_table_8 [id=5a8a0451cdf0479c94edf1d94e77fb83]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:37.414009 18813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51ef0be93c914e258635e13f3cca79aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:37.415084 19077 tablet_service.cc:1467] Processing CreateTablet for tablet 21af917743444f2db932636a78f33512 (DEFAULT_TABLE table=rebalance_test_table_8 [id=5a8a0451cdf0479c94edf1d94e77fb83]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:37.415511 19077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21af917743444f2db932636a78f33512. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:37.415644 18811 tablet_service.cc:1467] Processing CreateTablet for tablet 21af917743444f2db932636a78f33512 (DEFAULT_TABLE table=rebalance_test_table_8 [id=5a8a0451cdf0479c94edf1d94e77fb83]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:37.416049 18811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21af917743444f2db932636a78f33512. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:37.416164 19076 tablet_service.cc:1467] Processing CreateTablet for tablet db56136f2dbb429098d4d78b15a3a439 (DEFAULT_TABLE table=rebalance_test_table_8 [id=5a8a0451cdf0479c94edf1d94e77fb83]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:37.416541 19076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db56136f2dbb429098d4d78b15a3a439. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:37.422593 18681 tablet_service.cc:1467] Processing CreateTablet for tablet db56136f2dbb429098d4d78b15a3a439 (DEFAULT_TABLE table=rebalance_test_table_8 [id=5a8a0451cdf0479c94edf1d94e77fb83]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:37.423043 18681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db56136f2dbb429098d4d78b15a3a439. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:37.433861 20383 tablet_bootstrap.cc:492] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06: Bootstrap starting.
I20240502 14:26:37.435756 20383 tablet_bootstrap.cc:654] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:37.452822 20382 tablet_bootstrap.cc:492] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8: Bootstrap starting.
I20240502 14:26:37.453629 20385 tablet_bootstrap.cc:492] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b: Bootstrap starting.
I20240502 14:26:37.455830 20381 tablet_bootstrap.cc:492] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4: Bootstrap starting.
I20240502 14:26:37.455951 20382 tablet_bootstrap.cc:654] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:37.457211 20385 tablet_bootstrap.cc:654] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:37.458498 20381 tablet_bootstrap.cc:654] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:37.460443 20383 tablet_bootstrap.cc:492] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06: No bootstrap required, opened a new log
I20240502 14:26:37.460815 20383 ts_tablet_manager.cc:1367] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.009s
I20240502 14:26:37.462683 20385 tablet_bootstrap.cc:492] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b: No bootstrap required, opened a new log
I20240502 14:26:37.462575 20383 raft_consensus.cc:348] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:37.462927 20382 tablet_bootstrap.cc:492] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8: No bootstrap required, opened a new log
I20240502 14:26:37.462994 20385 ts_tablet_manager.cc:1367] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20240502 14:26:37.463069 20383 raft_consensus.cc:374] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:37.463110 20382 ts_tablet_manager.cc:1367] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240502 14:26:37.463308 20383 raft_consensus.cc:729] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Initialized, Role: FOLLOWER
I20240502 14:26:37.463820 20383 consensus_queue.cc:260] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:37.463923 20382 raft_consensus.cc:348] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:37.463924 20385 raft_consensus.cc:348] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:37.464129 20385 raft_consensus.cc:374] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:37.464129 20382 raft_consensus.cc:374] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:37.464226 20382 raft_consensus.cc:729] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Initialized, Role: FOLLOWER
I20240502 14:26:37.464255 20385 raft_consensus.cc:729] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Initialized, Role: FOLLOWER
I20240502 14:26:37.464531 20382 consensus_queue.cc:260] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:37.464519 20385 consensus_queue.cc:260] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:37.465139 20382 ts_tablet_manager.cc:1398] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:26:37.465687 20385 ts_tablet_manager.cc:1398] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:26:37.465987 20383 ts_tablet_manager.cc:1398] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:26:37.466048 20385 tablet_bootstrap.cc:492] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b: Bootstrap starting.
I20240502 14:26:37.466393 20383 tablet_bootstrap.cc:492] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06: Bootstrap starting.
I20240502 14:26:37.467839 20385 tablet_bootstrap.cc:654] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:37.468312 20383 tablet_bootstrap.cc:654] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:37.469753 20382 tablet_bootstrap.cc:492] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8: Bootstrap starting.
I20240502 14:26:37.472952 20382 tablet_bootstrap.cc:654] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:37.477555 20382 tablet_bootstrap.cc:492] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8: No bootstrap required, opened a new log
I20240502 14:26:37.477771 20382 ts_tablet_manager.cc:1367] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240502 14:26:37.478524 20382 raft_consensus.cc:348] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:37.478803 20382 raft_consensus.cc:374] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:37.478909 20382 raft_consensus.cc:729] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Initialized, Role: FOLLOWER
I20240502 14:26:37.478924 20381 tablet_bootstrap.cc:492] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4: No bootstrap required, opened a new log
I20240502 14:26:37.479130 20381 ts_tablet_manager.cc:1367] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.010s
I20240502 14:26:37.479233 20383 tablet_bootstrap.cc:492] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06: No bootstrap required, opened a new log
I20240502 14:26:37.479271 20382 consensus_queue.cc:260] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:37.479458 20383 ts_tablet_manager.cc:1367] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20240502 14:26:37.479866 20382 ts_tablet_manager.cc:1398] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:26:37.480356 20383 raft_consensus.cc:348] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:37.480729 20383 raft_consensus.cc:374] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:37.480871 20383 raft_consensus.cc:729] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Initialized, Role: FOLLOWER
I20240502 14:26:37.481308 20383 consensus_queue.cc:260] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:37.481272 20381 raft_consensus.cc:348] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:37.481783 20381 raft_consensus.cc:374] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:37.481930 20381 raft_consensus.cc:729] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Initialized, Role: FOLLOWER
I20240502 14:26:37.482945 20383 ts_tablet_manager.cc:1398] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:26:37.482224 20381 consensus_queue.cc:260] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:37.485060 20381 ts_tablet_manager.cc:1398] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:26:37.485869 20381 tablet_bootstrap.cc:492] T 21af917743444f2db932636a78f33512 P fec04c50dc5b49878ffa3c3873df32e4: Bootstrap starting.
I20240502 14:26:37.488667 20381 tablet_bootstrap.cc:654] T 21af917743444f2db932636a78f33512 P fec04c50dc5b49878ffa3c3873df32e4: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:37.490559 20381 tablet_bootstrap.cc:492] T 21af917743444f2db932636a78f33512 P fec04c50dc5b49878ffa3c3873df32e4: No bootstrap required, opened a new log
I20240502 14:26:37.490733 20381 ts_tablet_manager.cc:1367] T 21af917743444f2db932636a78f33512 P fec04c50dc5b49878ffa3c3873df32e4: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240502 14:26:37.491307 20381 raft_consensus.cc:348] T 21af917743444f2db932636a78f33512 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:37.491495 20381 raft_consensus.cc:374] T 21af917743444f2db932636a78f33512 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:37.491565 20381 raft_consensus.cc:729] T 21af917743444f2db932636a78f33512 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Initialized, Role: FOLLOWER
I20240502 14:26:37.491801 20381 consensus_queue.cc:260] T 21af917743444f2db932636a78f33512 P fec04c50dc5b49878ffa3c3873df32e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:37.492233 20381 ts_tablet_manager.cc:1398] T 21af917743444f2db932636a78f33512 P fec04c50dc5b49878ffa3c3873df32e4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:26:37.492813 20385 tablet_bootstrap.cc:492] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b: No bootstrap required, opened a new log
I20240502 14:26:37.492978 20385 ts_tablet_manager.cc:1367] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.001s
I20240502 14:26:37.493507 20381 tablet_bootstrap.cc:492] T 51ef0be93c914e258635e13f3cca79aa P fec04c50dc5b49878ffa3c3873df32e4: Bootstrap starting.
I20240502 14:26:37.493912 20385 raft_consensus.cc:348] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:37.494323 20385 raft_consensus.cc:374] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:37.494522 20385 raft_consensus.cc:729] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Initialized, Role: FOLLOWER
I20240502 14:26:37.494881 20385 consensus_queue.cc:260] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:37.495381 20381 tablet_bootstrap.cc:654] T 51ef0be93c914e258635e13f3cca79aa P fec04c50dc5b49878ffa3c3873df32e4: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:37.496551 20385 ts_tablet_manager.cc:1398] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:26:37.500221 20381 tablet_bootstrap.cc:492] T 51ef0be93c914e258635e13f3cca79aa P fec04c50dc5b49878ffa3c3873df32e4: No bootstrap required, opened a new log
I20240502 14:26:37.500376 20381 ts_tablet_manager.cc:1367] T 51ef0be93c914e258635e13f3cca79aa P fec04c50dc5b49878ffa3c3873df32e4: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20240502 14:26:37.501324 20381 raft_consensus.cc:348] T 51ef0be93c914e258635e13f3cca79aa P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:37.501495 20381 raft_consensus.cc:374] T 51ef0be93c914e258635e13f3cca79aa P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:37.501578 20381 raft_consensus.cc:729] T 51ef0be93c914e258635e13f3cca79aa P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Initialized, Role: FOLLOWER
I20240502 14:26:37.501801 20381 consensus_queue.cc:260] T 51ef0be93c914e258635e13f3cca79aa P fec04c50dc5b49878ffa3c3873df32e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:37.502455 20381 ts_tablet_manager.cc:1398] T 51ef0be93c914e258635e13f3cca79aa P fec04c50dc5b49878ffa3c3873df32e4: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 14:26:37.520628 20284 raft_consensus.cc:482] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:37.520809 20284 raft_consensus.cc:504] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:37.521595 20284 leader_election.cc:288] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409), fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
I20240502 14:26:37.522101 18965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ef0be93c914e258635e13f3cca79aa" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06" is_pre_election: true
I20240502 14:26:37.522140 18832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ef0be93c914e258635e13f3cca79aa" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec04c50dc5b49878ffa3c3873df32e4" is_pre_election: true
I20240502 14:26:37.522370 18965 raft_consensus.cc:2454] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2db304b538e14857b7b1cc272d0db94b in term 0.
I20240502 14:26:37.522387 18832 raft_consensus.cc:2454] T 51ef0be93c914e258635e13f3cca79aa P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2db304b538e14857b7b1cc272d0db94b in term 0.
I20240502 14:26:37.522825 18634 leader_election.cc:302] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b, fec04c50dc5b49878ffa3c3873df32e4; no voters:
I20240502 14:26:37.523082 20284 raft_consensus.cc:2781] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:37.523206 20284 raft_consensus.cc:482] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:37.523270 20284 raft_consensus.cc:3037] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:37.523948 20185 raft_consensus.cc:482] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:37.524096 20185 raft_consensus.cc:504] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:37.524770 20185 leader_election.cc:288] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009), 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:26:37.525303 18832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21af917743444f2db932636a78f33512" candidate_uuid: "c902805060df4df9b5599d0e94b2dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec04c50dc5b49878ffa3c3873df32e4" is_pre_election: true
I20240502 14:26:37.525385 19096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21af917743444f2db932636a78f33512" candidate_uuid: "c902805060df4df9b5599d0e94b2dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b2e79dc424a6f80141180f072f2d8" is_pre_election: true
I20240502 14:26:37.525498 18832 raft_consensus.cc:2454] T 21af917743444f2db932636a78f33512 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c902805060df4df9b5599d0e94b2dc06 in term 0.
I20240502 14:26:37.525664 19096 raft_consensus.cc:2454] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c902805060df4df9b5599d0e94b2dc06 in term 0.
I20240502 14:26:37.525689 20284 raft_consensus.cc:504] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:37.525890 18897 leader_election.cc:302] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c902805060df4df9b5599d0e94b2dc06, fec04c50dc5b49878ffa3c3873df32e4; no voters:
I20240502 14:26:37.526163 20185 raft_consensus.cc:2781] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:37.526273 20185 raft_consensus.cc:482] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:37.526304 20284 leader_election.cc:288] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 election: Requested vote from peers c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409), fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
I20240502 14:26:37.526381 20185 raft_consensus.cc:3037] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:37.526850 18965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ef0be93c914e258635e13f3cca79aa" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06"
I20240502 14:26:37.527021 18965 raft_consensus.cc:3037] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:37.528612 20185 raft_consensus.cc:504] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:37.528963 18965 raft_consensus.cc:2454] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2db304b538e14857b7b1cc272d0db94b in term 1.
I20240502 14:26:37.529305 20185 leader_election.cc:288] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 1 election: Requested vote from peers fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009), 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:26:37.529775 18832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21af917743444f2db932636a78f33512" candidate_uuid: "c902805060df4df9b5599d0e94b2dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec04c50dc5b49878ffa3c3873df32e4"
I20240502 14:26:37.529826 19096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21af917743444f2db932636a78f33512" candidate_uuid: "c902805060df4df9b5599d0e94b2dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b2e79dc424a6f80141180f072f2d8"
I20240502 14:26:37.529906 18637 leader_election.cc:302] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b, c902805060df4df9b5599d0e94b2dc06; no voters:
I20240502 14:26:37.529984 18832 raft_consensus.cc:3037] T 21af917743444f2db932636a78f33512 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:37.530033 19096 raft_consensus.cc:3037] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:37.530380 18833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ef0be93c914e258635e13f3cca79aa" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec04c50dc5b49878ffa3c3873df32e4"
I20240502 14:26:37.530573 18833 raft_consensus.cc:3037] T 51ef0be93c914e258635e13f3cca79aa P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:37.532450 20284 raft_consensus.cc:2781] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:26:37.532681 20284 raft_consensus.cc:686] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Becoming Leader. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Running, Role: LEADER
I20240502 14:26:37.532742 19096 raft_consensus.cc:2454] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c902805060df4df9b5599d0e94b2dc06 in term 1.
I20240502 14:26:37.533174 18833 raft_consensus.cc:2454] T 51ef0be93c914e258635e13f3cca79aa P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2db304b538e14857b7b1cc272d0db94b in term 1.
I20240502 14:26:37.533218 18900 leader_election.cc:302] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 007b2e79dc424a6f80141180f072f2d8, c902805060df4df9b5599d0e94b2dc06; no voters:
I20240502 14:26:37.533073 20284 consensus_queue.cc:237] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:37.533632 20185 raft_consensus.cc:2781] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:26:37.533908 18832 raft_consensus.cc:2454] T 21af917743444f2db932636a78f33512 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c902805060df4df9b5599d0e94b2dc06 in term 1.
I20240502 14:26:37.534219 20185 raft_consensus.cc:686] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Becoming Leader. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Running, Role: LEADER
I20240502 14:26:37.534605 20185 consensus_queue.cc:237] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:37.537755 18553 catalog_manager.cc:5484] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2db304b538e14857b7b1cc272d0db94b (127.5.107.65). New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:26:37.538923 18558 catalog_manager.cc:5484] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 reported cstate change: term changed from 0 to 1, leader changed from <none> to c902805060df4df9b5599d0e94b2dc06 (127.5.107.67). New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:26:37.843621 19984 raft_consensus.cc:482] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:37.843852 19984 raft_consensus.cc:504] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:37.844305 20284 raft_consensus.cc:482] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:37.844514 20284 raft_consensus.cc:504] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:37.844651 19984 leader_election.cc:288] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009), 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:26:37.844974 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db56136f2dbb429098d4d78b15a3a439" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b" is_pre_election: true
I20240502 14:26:37.845255 20284 leader_election.cc:288] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463), fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
I20240502 14:26:37.845257 18701 raft_consensus.cc:2454] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 0.
I20240502 14:26:37.845481 18832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db56136f2dbb429098d4d78b15a3a439" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec04c50dc5b49878ffa3c3873df32e4" is_pre_election: true
I20240502 14:26:37.845817 18833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db56136f2dbb429098d4d78b15a3a439" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec04c50dc5b49878ffa3c3873df32e4" is_pre_election: true
I20240502 14:26:37.845880 18832 raft_consensus.cc:2454] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 0.
I20240502 14:26:37.846010 19030 leader_election.cc:302] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 007b2e79dc424a6f80141180f072f2d8, 2db304b538e14857b7b1cc272d0db94b; no voters:
I20240502 14:26:37.846168 19096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db56136f2dbb429098d4d78b15a3a439" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b2e79dc424a6f80141180f072f2d8" is_pre_election: true
I20240502 14:26:37.846423 19096 raft_consensus.cc:2454] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2db304b538e14857b7b1cc272d0db94b in term 0.
I20240502 14:26:37.846613 19984 raft_consensus.cc:2781] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:37.846724 19984 raft_consensus.cc:482] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:37.846844 19984 raft_consensus.cc:3037] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:37.846832 18637 leader_election.cc:302] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 007b2e79dc424a6f80141180f072f2d8, 2db304b538e14857b7b1cc272d0db94b; no voters: fec04c50dc5b49878ffa3c3873df32e4
I20240502 14:26:37.847122 20284 raft_consensus.cc:2781] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:37.847208 20284 raft_consensus.cc:482] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:37.847306 20284 raft_consensus.cc:3037] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:37.849301 19984 raft_consensus.cc:504] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:37.849390 20284 raft_consensus.cc:504] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:37.850430 20284 leader_election.cc:288] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 election: Requested vote from peers 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463), fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
I20240502 14:26:37.850437 19096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db56136f2dbb429098d4d78b15a3a439" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b2e79dc424a6f80141180f072f2d8"
I20240502 14:26:37.850515 19984 leader_election.cc:288] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 election: Requested vote from peers fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009), 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:26:37.850519 18833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db56136f2dbb429098d4d78b15a3a439" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec04c50dc5b49878ffa3c3873df32e4"
I20240502 14:26:37.850736 19096 raft_consensus.cc:2379] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2db304b538e14857b7b1cc272d0db94b in current term 1: Already voted for candidate 007b2e79dc424a6f80141180f072f2d8 in this term.
I20240502 14:26:37.850740 18833 raft_consensus.cc:3037] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:37.851049 18832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db56136f2dbb429098d4d78b15a3a439" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec04c50dc5b49878ffa3c3873df32e4"
I20240502 14:26:37.852715 18833 raft_consensus.cc:2454] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2db304b538e14857b7b1cc272d0db94b in term 1.
I20240502 14:26:37.853101 18634 leader_election.cc:302] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b, fec04c50dc5b49878ffa3c3873df32e4; no voters: 007b2e79dc424a6f80141180f072f2d8
I20240502 14:26:37.853461 20284 raft_consensus.cc:2781] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:26:37.853456 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db56136f2dbb429098d4d78b15a3a439" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b"
I20240502 14:26:37.853677 20284 raft_consensus.cc:686] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Becoming Leader. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Running, Role: LEADER
I20240502 14:26:37.853928 20284 consensus_queue.cc:237] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:26:37.855705 19030 leader_election.cc:302] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 007b2e79dc424a6f80141180f072f2d8; no voters: 2db304b538e14857b7b1cc272d0db94b, fec04c50dc5b49878ffa3c3873df32e4
I20240502 14:26:37.856017 19984 raft_consensus.cc:2726] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 14:26:37.857292 18558 catalog_manager.cc:5484] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2db304b538e14857b7b1cc272d0db94b (127.5.107.65). New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:26:37.987991 20284 consensus_queue.cc:1035] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:37.993850 20284 consensus_queue.cc:1035] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:38.122140 20185 consensus_queue.cc:1035] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:38.127568 20185 consensus_queue.cc:1035] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:26:38.258775 20398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:26:38.259119 20398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:26:38.267489 20398 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240502 14:26:38.326264 20284 consensus_queue.cc:1035] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:38.347947 20284 consensus_queue.cc:1035] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 285.202 ms
time per row: 0.142601 ms
I20240502 14:26:38.706682 18560 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48228:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:26:38.715813 18681 tablet_service.cc:1467] Processing CreateTablet for tablet 1ecc6539b85d40e49df7ffc5f93cb801 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a9acd50a500c4f84a41c8d9ba23ae431]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:38.715978 18945 tablet_service.cc:1467] Processing CreateTablet for tablet 1ecc6539b85d40e49df7ffc5f93cb801 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a9acd50a500c4f84a41c8d9ba23ae431]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:38.716320 18681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ecc6539b85d40e49df7ffc5f93cb801. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:38.716398 18945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ecc6539b85d40e49df7ffc5f93cb801. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:38.717434 18812 tablet_service.cc:1467] Processing CreateTablet for tablet 2c2afaffde96462eacd0d32bbdec23b1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a9acd50a500c4f84a41c8d9ba23ae431]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:38.717873 18812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c2afaffde96462eacd0d32bbdec23b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:38.721691 19077 tablet_service.cc:1467] Processing CreateTablet for tablet 1ecc6539b85d40e49df7ffc5f93cb801 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a9acd50a500c4f84a41c8d9ba23ae431]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:38.722138 19077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ecc6539b85d40e49df7ffc5f93cb801. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:38.722697 20461 tablet_bootstrap.cc:492] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b: Bootstrap starting.
I20240502 14:26:38.724332 20461 tablet_bootstrap.cc:654] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:38.725003 19076 tablet_service.cc:1467] Processing CreateTablet for tablet 246a8425400f4890a5264c4ae8a627c5 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a9acd50a500c4f84a41c8d9ba23ae431]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:38.725471 19076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 246a8425400f4890a5264c4ae8a627c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:38.730891 18680 tablet_service.cc:1467] Processing CreateTablet for tablet 2c2afaffde96462eacd0d32bbdec23b1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a9acd50a500c4f84a41c8d9ba23ae431]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:38.731556 18680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c2afaffde96462eacd0d32bbdec23b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:38.731809 18944 tablet_service.cc:1467] Processing CreateTablet for tablet 246a8425400f4890a5264c4ae8a627c5 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a9acd50a500c4f84a41c8d9ba23ae431]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:38.732398 18944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 246a8425400f4890a5264c4ae8a627c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:38.731830 18943 tablet_service.cc:1467] Processing CreateTablet for tablet 2c2afaffde96462eacd0d32bbdec23b1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a9acd50a500c4f84a41c8d9ba23ae431]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:38.733347 18943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c2afaffde96462eacd0d32bbdec23b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:38.742734 18681 tablet_service.cc:1467] Processing CreateTablet for tablet 246a8425400f4890a5264c4ae8a627c5 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a9acd50a500c4f84a41c8d9ba23ae431]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:26:38.752580 18681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 246a8425400f4890a5264c4ae8a627c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:38.753993 20461 tablet_bootstrap.cc:492] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b: No bootstrap required, opened a new log
I20240502 14:26:38.754240 20461 ts_tablet_manager.cc:1367] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b: Time spent bootstrapping tablet: real 0.032s user 0.019s sys 0.010s
I20240502 14:26:38.755216 20461 raft_consensus.cc:348] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:38.755633 20461 raft_consensus.cc:374] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:38.755801 20461 raft_consensus.cc:729] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Initialized, Role: FOLLOWER
I20240502 14:26:38.756186 20461 consensus_queue.cc:260] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:38.756939 20461 ts_tablet_manager.cc:1398] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:26:38.757596 20461 tablet_bootstrap.cc:492] T 2c2afaffde96462eacd0d32bbdec23b1 P 2db304b538e14857b7b1cc272d0db94b: Bootstrap starting.
I20240502 14:26:38.759745 20461 tablet_bootstrap.cc:654] T 2c2afaffde96462eacd0d32bbdec23b1 P 2db304b538e14857b7b1cc272d0db94b: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:38.765276 20284 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:38.765466 20284 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:38.766191 20284 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409), 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:26:38.767153 18965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ecc6539b85d40e49df7ffc5f93cb801" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06" is_pre_election: true
W20240502 14:26:38.767993 18637 leader_election.cc:341] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:26:38.768322 19095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ecc6539b85d40e49df7ffc5f93cb801" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b2e79dc424a6f80141180f072f2d8" is_pre_election: true
W20240502 14:26:38.768801 18637 leader_election.cc:341] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:26:38.768926 18637 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: 007b2e79dc424a6f80141180f072f2d8, c902805060df4df9b5599d0e94b2dc06
I20240502 14:26:38.769762 20284 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:26:38.773825 20464 tablet_bootstrap.cc:492] T 246a8425400f4890a5264c4ae8a627c5 P 007b2e79dc424a6f80141180f072f2d8: Bootstrap starting.
I20240502 14:26:38.773452 20465 tablet_bootstrap.cc:492] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06: Bootstrap starting.
I20240502 14:26:38.774143 20461 tablet_bootstrap.cc:492] T 2c2afaffde96462eacd0d32bbdec23b1 P 2db304b538e14857b7b1cc272d0db94b: No bootstrap required, opened a new log
I20240502 14:26:38.774447 20461 ts_tablet_manager.cc:1367] T 2c2afaffde96462eacd0d32bbdec23b1 P 2db304b538e14857b7b1cc272d0db94b: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.001s
I20240502 14:26:38.775509 20461 raft_consensus.cc:348] T 2c2afaffde96462eacd0d32bbdec23b1 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:38.775780 20461 raft_consensus.cc:374] T 2c2afaffde96462eacd0d32bbdec23b1 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:38.775905 20461 raft_consensus.cc:729] T 2c2afaffde96462eacd0d32bbdec23b1 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Initialized, Role: FOLLOWER
I20240502 14:26:38.776222 20461 consensus_queue.cc:260] T 2c2afaffde96462eacd0d32bbdec23b1 P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:38.776937 20461 ts_tablet_manager.cc:1398] T 2c2afaffde96462eacd0d32bbdec23b1 P 2db304b538e14857b7b1cc272d0db94b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:26:38.777383 20461 tablet_bootstrap.cc:492] T 246a8425400f4890a5264c4ae8a627c5 P 2db304b538e14857b7b1cc272d0db94b: Bootstrap starting.
I20240502 14:26:38.779630 20461 tablet_bootstrap.cc:654] T 246a8425400f4890a5264c4ae8a627c5 P 2db304b538e14857b7b1cc272d0db94b: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:38.782269 20462 tablet_bootstrap.cc:492] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4: Bootstrap starting.
I20240502 14:26:38.783339 20465 tablet_bootstrap.cc:654] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:38.783957 20464 tablet_bootstrap.cc:654] T 246a8425400f4890a5264c4ae8a627c5 P 007b2e79dc424a6f80141180f072f2d8: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:38.784057 20462 tablet_bootstrap.cc:654] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:38.786569 20461 tablet_bootstrap.cc:492] T 246a8425400f4890a5264c4ae8a627c5 P 2db304b538e14857b7b1cc272d0db94b: No bootstrap required, opened a new log
I20240502 14:26:38.786741 20461 ts_tablet_manager.cc:1367] T 246a8425400f4890a5264c4ae8a627c5 P 2db304b538e14857b7b1cc272d0db94b: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20240502 14:26:38.787566 20461 raft_consensus.cc:348] T 246a8425400f4890a5264c4ae8a627c5 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:38.787781 20461 raft_consensus.cc:374] T 246a8425400f4890a5264c4ae8a627c5 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:38.787864 20461 raft_consensus.cc:729] T 246a8425400f4890a5264c4ae8a627c5 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Initialized, Role: FOLLOWER
I20240502 14:26:38.788120 20461 consensus_queue.cc:260] T 246a8425400f4890a5264c4ae8a627c5 P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:38.788298 20465 tablet_bootstrap.cc:492] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06: No bootstrap required, opened a new log
I20240502 14:26:38.788703 20461 ts_tablet_manager.cc:1398] T 246a8425400f4890a5264c4ae8a627c5 P 2db304b538e14857b7b1cc272d0db94b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:26:38.789364 20465 ts_tablet_manager.cc:1367] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.002s
I20240502 14:26:38.790513 20465 raft_consensus.cc:348] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:38.790784 20465 raft_consensus.cc:374] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:38.790951 20465 raft_consensus.cc:729] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Initialized, Role: FOLLOWER
I20240502 14:26:38.792357 20464 tablet_bootstrap.cc:492] T 246a8425400f4890a5264c4ae8a627c5 P 007b2e79dc424a6f80141180f072f2d8: No bootstrap required, opened a new log
I20240502 14:26:38.792523 20464 ts_tablet_manager.cc:1367] T 246a8425400f4890a5264c4ae8a627c5 P 007b2e79dc424a6f80141180f072f2d8: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20240502 14:26:38.793339 20464 raft_consensus.cc:348] T 246a8425400f4890a5264c4ae8a627c5 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:38.793587 20464 raft_consensus.cc:374] T 246a8425400f4890a5264c4ae8a627c5 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:38.793728 20464 raft_consensus.cc:729] T 246a8425400f4890a5264c4ae8a627c5 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Initialized, Role: FOLLOWER
I20240502 14:26:38.793962 20464 consensus_queue.cc:260] T 246a8425400f4890a5264c4ae8a627c5 P 007b2e79dc424a6f80141180f072f2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:38.794515 20464 ts_tablet_manager.cc:1398] T 246a8425400f4890a5264c4ae8a627c5 P 007b2e79dc424a6f80141180f072f2d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:26:38.794982 20464 tablet_bootstrap.cc:492] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8: Bootstrap starting.
I20240502 14:26:38.795707 20462 tablet_bootstrap.cc:492] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4: No bootstrap required, opened a new log
I20240502 14:26:38.795908 20462 ts_tablet_manager.cc:1367] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20240502 14:26:38.796481 20464 tablet_bootstrap.cc:654] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:38.791337 20465 consensus_queue.cc:260] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:38.796813 20462 raft_consensus.cc:348] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:38.797011 20462 raft_consensus.cc:374] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:38.797117 20462 raft_consensus.cc:729] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Initialized, Role: FOLLOWER
I20240502 14:26:38.797452 20462 consensus_queue.cc:260] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:38.798981 20462 ts_tablet_manager.cc:1398] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:26:38.800551 20465 ts_tablet_manager.cc:1398] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240502 14:26:38.800750 20464 tablet_bootstrap.cc:492] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8: No bootstrap required, opened a new log
I20240502 14:26:38.800877 20465 tablet_bootstrap.cc:492] T 2c2afaffde96462eacd0d32bbdec23b1 P c902805060df4df9b5599d0e94b2dc06: Bootstrap starting.
I20240502 14:26:38.800920 20464 ts_tablet_manager.cc:1367] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 14:26:38.801967 20464 raft_consensus.cc:348] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:38.802261 20464 raft_consensus.cc:374] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:38.802382 20464 raft_consensus.cc:729] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Initialized, Role: FOLLOWER
I20240502 14:26:38.802747 20464 consensus_queue.cc:260] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:38.803334 20464 ts_tablet_manager.cc:1398] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:26:38.803411 20465 tablet_bootstrap.cc:654] T 2c2afaffde96462eacd0d32bbdec23b1 P c902805060df4df9b5599d0e94b2dc06: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:38.805794 20465 tablet_bootstrap.cc:492] T 2c2afaffde96462eacd0d32bbdec23b1 P c902805060df4df9b5599d0e94b2dc06: No bootstrap required, opened a new log
I20240502 14:26:38.806099 20465 ts_tablet_manager.cc:1367] T 2c2afaffde96462eacd0d32bbdec23b1 P c902805060df4df9b5599d0e94b2dc06: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240502 14:26:38.806830 20465 raft_consensus.cc:348] T 2c2afaffde96462eacd0d32bbdec23b1 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:38.807070 20465 raft_consensus.cc:374] T 2c2afaffde96462eacd0d32bbdec23b1 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:38.807153 20465 raft_consensus.cc:729] T 2c2afaffde96462eacd0d32bbdec23b1 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Initialized, Role: FOLLOWER
I20240502 14:26:38.807442 20465 consensus_queue.cc:260] T 2c2afaffde96462eacd0d32bbdec23b1 P c902805060df4df9b5599d0e94b2dc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:38.808382 20465 ts_tablet_manager.cc:1398] T 2c2afaffde96462eacd0d32bbdec23b1 P c902805060df4df9b5599d0e94b2dc06: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:26:38.808694 20465 tablet_bootstrap.cc:492] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06: Bootstrap starting.
I20240502 14:26:38.810248 20465 tablet_bootstrap.cc:654] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06: Neither blocks nor log segments found. Creating new log.
I20240502 14:26:38.814316 20465 tablet_bootstrap.cc:492] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06: No bootstrap required, opened a new log
I20240502 14:26:38.814510 20465 ts_tablet_manager.cc:1367] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240502 14:26:38.815168 20465 raft_consensus.cc:348] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:38.815316 20465 raft_consensus.cc:374] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:26:38.815382 20465 raft_consensus.cc:729] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Initialized, Role: FOLLOWER
I20240502 14:26:38.815555 20465 consensus_queue.cc:260] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:38.815953 20465 ts_tablet_manager.cc:1398] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:26:38.818306 20473 raft_consensus.cc:482] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:38.818509 20473 raft_consensus.cc:504] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:38.819536 20473 leader_election.cc:288] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749), c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:26:38.819814 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c2afaffde96462eacd0d32bbdec23b1" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b" is_pre_election: true
I20240502 14:26:38.819993 18701 raft_consensus.cc:2454] T 2c2afaffde96462eacd0d32bbdec23b1 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 0.
I20240502 14:26:38.820425 18766 leader_election.cc:302] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b, fec04c50dc5b49878ffa3c3873df32e4; no voters:
I20240502 14:26:38.820612 18963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c2afaffde96462eacd0d32bbdec23b1" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06" is_pre_election: true
I20240502 14:26:38.820791 20473 raft_consensus.cc:2781] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:38.820839 18963 raft_consensus.cc:2454] T 2c2afaffde96462eacd0d32bbdec23b1 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 0.
I20240502 14:26:38.820885 20473 raft_consensus.cc:482] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:38.820945 20473 raft_consensus.cc:3037] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:38.823041 20473 raft_consensus.cc:504] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:38.823594 20473 leader_election.cc:288] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 election: Requested vote from peers 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749), c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:26:38.824039 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c2afaffde96462eacd0d32bbdec23b1" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b"
I20240502 14:26:38.824121 18963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c2afaffde96462eacd0d32bbdec23b1" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06"
I20240502 14:26:38.824266 18701 raft_consensus.cc:3037] T 2c2afaffde96462eacd0d32bbdec23b1 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:38.824314 18963 raft_consensus.cc:3037] T 2c2afaffde96462eacd0d32bbdec23b1 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:38.827067 18701 raft_consensus.cc:2454] T 2c2afaffde96462eacd0d32bbdec23b1 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 1.
I20240502 14:26:38.827067 18963 raft_consensus.cc:2454] T 2c2afaffde96462eacd0d32bbdec23b1 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 1.
I20240502 14:26:38.827466 18766 leader_election.cc:302] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b, fec04c50dc5b49878ffa3c3873df32e4; no voters:
I20240502 14:26:38.827805 20473 raft_consensus.cc:2781] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:26:38.827946 20473 raft_consensus.cc:686] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Becoming Leader. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Running, Role: LEADER
I20240502 14:26:38.828241 20473 consensus_queue.cc:237] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:38.831116 18555 catalog_manager.cc:5484] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66). New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
I20240502 14:26:39.039166 20433 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:39.039332 20433 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:39.039932 20433 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749), c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:26:39.040211 20185 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:39.040509 18963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ecc6539b85d40e49df7ffc5f93cb801" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06" is_pre_election: true
I20240502 14:26:39.040519 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ecc6539b85d40e49df7ffc5f93cb801" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b" is_pre_election: true
I20240502 14:26:39.040462 20185 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:39.040764 18701 raft_consensus.cc:2454] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 0.
I20240502 14:26:39.040833 18963 raft_consensus.cc:2454] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 0.
I20240502 14:26:39.041164 20185 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749), 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:26:39.041363 19030 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 007b2e79dc424a6f80141180f072f2d8, 2db304b538e14857b7b1cc272d0db94b; no voters:
I20240502 14:26:39.041690 19095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ecc6539b85d40e49df7ffc5f93cb801" candidate_uuid: "c902805060df4df9b5599d0e94b2dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b2e79dc424a6f80141180f072f2d8" is_pre_election: true
I20240502 14:26:39.041908 20433 raft_consensus.cc:2781] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:39.042045 20433 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:39.042141 20433 raft_consensus.cc:3037] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:39.042148 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ecc6539b85d40e49df7ffc5f93cb801" candidate_uuid: "c902805060df4df9b5599d0e94b2dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b" is_pre_election: true
I20240502 14:26:39.042384 18701 raft_consensus.cc:2454] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c902805060df4df9b5599d0e94b2dc06 in term 0.
I20240502 14:26:39.042805 18898 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b, c902805060df4df9b5599d0e94b2dc06; no voters:
I20240502 14:26:39.043071 20185 raft_consensus.cc:2781] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:39.043162 20185 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:39.043285 20185 raft_consensus.cc:3037] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:39.044437 20433 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:39.044807 19095 raft_consensus.cc:2379] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c902805060df4df9b5599d0e94b2dc06 in current term 1: Already voted for candidate 007b2e79dc424a6f80141180f072f2d8 in this term.
I20240502 14:26:39.045454 20185 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:39.045524 18963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ecc6539b85d40e49df7ffc5f93cb801" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06"
I20240502 14:26:39.045418 20433 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 election: Requested vote from peers 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749), c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:26:39.045825 18963 raft_consensus.cc:2379] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 007b2e79dc424a6f80141180f072f2d8 in current term 1: Already voted for candidate c902805060df4df9b5599d0e94b2dc06 in this term.
I20240502 14:26:39.046025 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ecc6539b85d40e49df7ffc5f93cb801" candidate_uuid: "007b2e79dc424a6f80141180f072f2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b"
I20240502 14:26:39.046281 18701 raft_consensus.cc:3037] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:39.046279 20185 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 1 election: Requested vote from peers 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749), 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:26:39.046746 19095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ecc6539b85d40e49df7ffc5f93cb801" candidate_uuid: "c902805060df4df9b5599d0e94b2dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b2e79dc424a6f80141180f072f2d8"
I20240502 14:26:39.046854 18700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ecc6539b85d40e49df7ffc5f93cb801" candidate_uuid: "c902805060df4df9b5599d0e94b2dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b"
I20240502 14:26:39.046973 19095 raft_consensus.cc:2379] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c902805060df4df9b5599d0e94b2dc06 in current term 1: Already voted for candidate 007b2e79dc424a6f80141180f072f2d8 in this term.
I20240502 14:26:39.047519 18898 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c902805060df4df9b5599d0e94b2dc06; no voters: 007b2e79dc424a6f80141180f072f2d8, 2db304b538e14857b7b1cc272d0db94b
I20240502 14:26:39.047747 20185 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 14:26:39.048491 18701 raft_consensus.cc:2454] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 007b2e79dc424a6f80141180f072f2d8 in term 1.
I20240502 14:26:39.048894 19030 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 007b2e79dc424a6f80141180f072f2d8, 2db304b538e14857b7b1cc272d0db94b; no voters: c902805060df4df9b5599d0e94b2dc06
I20240502 14:26:39.049257 20433 raft_consensus.cc:2781] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:26:39.049639 20433 raft_consensus.cc:686] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Becoming Leader. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Running, Role: LEADER
I20240502 14:26:39.050012 20433 consensus_queue.cc:237] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:26:39.053037 18555 catalog_manager.cc:5484] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68). New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:26:39.214216 20185 raft_consensus.cc:482] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:26:39.214426 20185 raft_consensus.cc:504] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:39.215252 20185 leader_election.cc:288] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749), 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:26:39.215819 18700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246a8425400f4890a5264c4ae8a627c5" candidate_uuid: "c902805060df4df9b5599d0e94b2dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b" is_pre_election: true
I20240502 14:26:39.215831 19095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246a8425400f4890a5264c4ae8a627c5" candidate_uuid: "c902805060df4df9b5599d0e94b2dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b2e79dc424a6f80141180f072f2d8" is_pre_election: true
I20240502 14:26:39.216053 18700 raft_consensus.cc:2454] T 246a8425400f4890a5264c4ae8a627c5 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c902805060df4df9b5599d0e94b2dc06 in term 0.
I20240502 14:26:39.216071 19095 raft_consensus.cc:2454] T 246a8425400f4890a5264c4ae8a627c5 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c902805060df4df9b5599d0e94b2dc06 in term 0.
I20240502 14:26:39.216495 18898 leader_election.cc:302] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b, c902805060df4df9b5599d0e94b2dc06; no voters:
I20240502 14:26:39.216778 20185 raft_consensus.cc:2781] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:26:39.216903 20185 raft_consensus.cc:482] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:26:39.216996 20185 raft_consensus.cc:3037] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:39.218880 20185 raft_consensus.cc:504] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:39.219473 20185 leader_election.cc:288] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 1 election: Requested vote from peers 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749), 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:26:39.220007 18700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246a8425400f4890a5264c4ae8a627c5" candidate_uuid: "c902805060df4df9b5599d0e94b2dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db304b538e14857b7b1cc272d0db94b"
I20240502 14:26:39.220067 19095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246a8425400f4890a5264c4ae8a627c5" candidate_uuid: "c902805060df4df9b5599d0e94b2dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b2e79dc424a6f80141180f072f2d8"
I20240502 14:26:39.220216 18700 raft_consensus.cc:3037] T 246a8425400f4890a5264c4ae8a627c5 P 2db304b538e14857b7b1cc272d0db94b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:39.220269 19095 raft_consensus.cc:3037] T 246a8425400f4890a5264c4ae8a627c5 P 007b2e79dc424a6f80141180f072f2d8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:26:39.222646 19095 raft_consensus.cc:2454] T 246a8425400f4890a5264c4ae8a627c5 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c902805060df4df9b5599d0e94b2dc06 in term 1.
I20240502 14:26:39.222649 18700 raft_consensus.cc:2454] T 246a8425400f4890a5264c4ae8a627c5 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c902805060df4df9b5599d0e94b2dc06 in term 1.
I20240502 14:26:39.223033 18900 leader_election.cc:302] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 007b2e79dc424a6f80141180f072f2d8, c902805060df4df9b5599d0e94b2dc06; no voters:
I20240502 14:26:39.223356 20185 raft_consensus.cc:2781] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:26:39.223747 20185 raft_consensus.cc:686] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Becoming Leader. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Running, Role: LEADER
I20240502 14:26:39.224011 20185 consensus_queue.cc:237] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:26:39.226984 18555 catalog_manager.cc:5484] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 reported cstate change: term changed from 0 to 1, leader changed from <none> to c902805060df4df9b5599d0e94b2dc06 (127.5.107.67). New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
I20240502 14:26:39.377712 20473 consensus_queue.cc:1035] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:39.383138 20482 consensus_queue.cc:1035] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:39.518278 20476 consensus_queue.cc:1035] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:26:39.524726 20433 consensus_queue.cc:1035] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:26:39.716765 20480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:26:39.717141 20480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 14:26:39.725555 20478 consensus_queue.cc:1035] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:26:39.726141 20480 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:26:39.727937 20478 consensus_queue.cc:1035] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
INSERT report
rows total: 2000
time total: 286.689 ms
time per row: 0.143345 ms
I20240502 14:26:40.167757 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.69:41119
--local_ip_for_outbound_sockets=127.5.107.69
--tserver_master_addrs=127.5.107.126:33551
--webserver_port=46705
--webserver_interface=127.5.107.69
--builtin_ntp_servers=127.5.107.84:43287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 14:26:40.527577 20543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:26:40.527882 20543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:26:40.527987 20543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:26:40.535624 20543 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:26:40.535753 20543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:26:40.535864 20543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.69
I20240502 14:26:40.545861 20543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:43287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.69:41119
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.5.107.69
--webserver_port=46705
--tserver_master_addrs=127.5.107.126:33551
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.69
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:26:40.547305 20543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:26:40.549120 20543 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:26:40.563921 20549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:26:40.564705 20550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:26:40.565124 20552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:26:40.566601 20543 server_base.cc:1034] running on GCE node
I20240502 14:26:40.567241 20543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:26:40.568244 20543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:26:40.569465 20543 hybrid_clock.cc:648] HybridClock initialized: now 1714660000569409 us; error 51 us; skew 500 ppm
I20240502 14:26:40.569768 20543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:26:40.572108 20543 webserver.cc:460] Webserver started at http://127.5.107.69:46705/ using document root <none> and password file <none>
I20240502 14:26:40.572880 20543 fs_manager.cc:359] Metadata directory not provided
I20240502 14:26:40.573017 20543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:26:40.579246 20543 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 14:26:40.582278 20558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:40.583775 20543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 14:26:40.583931 20543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "a834edffbcb2469aae598b069aab0681"
format_stamp: "Formatted at 2024-05-02 14:26:22 on dist-test-slave-2cw1"
I20240502 14:26:40.584776 20543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:26:40.599927 20543 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:26:40.600975 20543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:26:40.601328 20543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:26:40.602420 20543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:26:40.603932 20543 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:26:40.604005 20543 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:40.604105 20543 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:26:40.604151 20543 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:40.637778 20543 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.69:41119
I20240502 14:26:40.637810 20671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.69:41119 every 8 connection(s)
I20240502 14:26:40.639528 20543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:26:40.647146 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 20543
I20240502 14:26:40.647812 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.70:46021
--local_ip_for_outbound_sockets=127.5.107.70
--tserver_master_addrs=127.5.107.126:33551
--webserver_port=32901
--webserver_interface=127.5.107.70
--builtin_ntp_servers=127.5.107.84:43287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:26:40.653158 20672 heartbeater.cc:347] Connected to a master server at 127.5.107.126:33551
I20240502 14:26:40.653518 20672 heartbeater.cc:464] Registering TS with master...
I20240502 14:26:40.654263 20672 heartbeater.cc:511] Master 127.5.107.126:33551 requested a full tablet report, sending...
I20240502 14:26:40.655973 18551 ts_manager.cc:194] Re-registered known tserver with Master: a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119)
I20240502 14:26:40.657208 18551 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.69:40139
W20240502 14:26:41.022186 20676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:26:41.022522 20676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:26:41.022672 20676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:26:41.030638 20676 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:26:41.030803 20676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:26:41.030941 20676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.70
I20240502 14:26:41.040834 20676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:43287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.70:46021
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.5.107.70
--webserver_port=32901
--tserver_master_addrs=127.5.107.126:33551
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.70
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:26:41.042378 20676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:26:41.044299 20676 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:26:41.058161 20681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:26:41.058648 20682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:26:41.059159 20684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:26:41.059872 20676 server_base.cc:1034] running on GCE node
I20240502 14:26:41.060575 20676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:26:41.061699 20676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:26:41.062919 20676 hybrid_clock.cc:648] HybridClock initialized: now 1714660001062835 us; error 59 us; skew 500 ppm
I20240502 14:26:41.063248 20676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:26:41.065683 20676 webserver.cc:460] Webserver started at http://127.5.107.70:32901/ using document root <none> and password file <none>
I20240502 14:26:41.066445 20676 fs_manager.cc:359] Metadata directory not provided
I20240502 14:26:41.066573 20676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:26:41.072005 20676 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:26:41.074985 20690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:41.076452 20676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 14:26:41.076618 20676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0"
format_stamp: "Formatted at 2024-05-02 14:26:22 on dist-test-slave-2cw1"
I20240502 14:26:41.077692 20676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:26:41.097867 20676 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:26:41.098783 20676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:26:41.099087 20676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:26:41.100126 20676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:26:41.101960 20676 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:26:41.102051 20676 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:41.102190 20676 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:26:41.102252 20676 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:41.137692 20676 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.70:46021
I20240502 14:26:41.137755 20805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.70:46021 every 8 connection(s)
I20240502 14:26:41.139438 20676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 14:26:41.148972 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 20676
I20240502 14:26:41.149816 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.71:39195
--local_ip_for_outbound_sockets=127.5.107.71
--tserver_master_addrs=127.5.107.126:33551
--webserver_port=46407
--webserver_interface=127.5.107.71
--builtin_ntp_servers=127.5.107.84:43287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:26:41.153477 20806 heartbeater.cc:347] Connected to a master server at 127.5.107.126:33551
I20240502 14:26:41.153762 20806 heartbeater.cc:464] Registering TS with master...
I20240502 14:26:41.154748 20806 heartbeater.cc:511] Master 127.5.107.126:33551 requested a full tablet report, sending...
I20240502 14:26:41.156642 18551 ts_manager.cc:194] Re-registered known tserver with Master: a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021)
I20240502 14:26:41.157641 18551 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.70:34087
W20240502 14:26:41.510294 20809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:26:41.510607 20809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:26:41.510695 20809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:26:41.518422 20809 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:26:41.518555 20809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:26:41.518718 20809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.71
I20240502 14:26:41.529929 20809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:43287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.71:39195
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.5.107.71
--webserver_port=46407
--tserver_master_addrs=127.5.107.126:33551
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.71
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:26:41.531467 20809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:26:41.533265 20809 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:26:41.546661 20815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:26:41.547207 20818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:26:41.547722 20816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:26:41.548422 20809 server_base.cc:1034] running on GCE node
I20240502 14:26:41.548995 20809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:26:41.549983 20809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:26:41.551175 20809 hybrid_clock.cc:648] HybridClock initialized: now 1714660001551117 us; error 51 us; skew 500 ppm
I20240502 14:26:41.551503 20809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:26:41.553879 20809 webserver.cc:460] Webserver started at http://127.5.107.71:46407/ using document root <none> and password file <none>
I20240502 14:26:41.554642 20809 fs_manager.cc:359] Metadata directory not provided
I20240502 14:26:41.554776 20809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:26:41.560163 20809 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20240502 14:26:41.563115 20824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:41.564538 20809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 14:26:41.564713 20809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "7ab6d395b30d4afb9821b61c9454430b"
format_stamp: "Formatted at 2024-05-02 14:26:22 on dist-test-slave-2cw1"
I20240502 14:26:41.565553 20809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:26:41.605285 20809 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:26:41.606137 20809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:26:41.606391 20809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:26:41.607311 20809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:26:41.608954 20809 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:26:41.609028 20809 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:41.609122 20809 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:26:41.609169 20809 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:41.651436 20809 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.71:39195
I20240502 14:26:41.651512 20938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.71:39195 every 8 connection(s)
I20240502 14:26:41.652911 20809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 14:26:41.659793 20672 heartbeater.cc:503] Master 127.5.107.126:33551 was elected leader, sending a full tablet report...
I20240502 14:26:41.660900 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 20809
I20240502 14:26:41.661712 5549 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
/tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.107.72:35871
--local_ip_for_outbound_sockets=127.5.107.72
--tserver_master_addrs=127.5.107.126:33551
--webserver_port=36821
--webserver_interface=127.5.107.72
--builtin_ntp_servers=127.5.107.84:43287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:26:41.667150 20939 heartbeater.cc:347] Connected to a master server at 127.5.107.126:33551
I20240502 14:26:41.667452 20939 heartbeater.cc:464] Registering TS with master...
I20240502 14:26:41.668191 20939 heartbeater.cc:511] Master 127.5.107.126:33551 requested a full tablet report, sending...
I20240502 14:26:41.669701 18551 ts_manager.cc:194] Re-registered known tserver with Master: 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
I20240502 14:26:41.670593 18551 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.71:43139
W20240502 14:26:42.035650 20943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:26:42.035957 20943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:26:42.036085 20943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:26:42.044032 20943 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:26:42.044168 20943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:26:42.044333 20943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.107.72
I20240502 14:26:42.054097 20943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.107.84:43287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.107.72:35871
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.5.107.72
--webserver_port=36821
--tserver_master_addrs=127.5.107.126:33551
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.107.72
--log_dir=/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:26:42.055495 20943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:26:42.057173 20943 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:26:42.069774 20952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:26:42.070255 20954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:26:42.070900 20951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:26:42.071470 20943 server_base.cc:1034] running on GCE node
I20240502 14:26:42.072121 20943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:26:42.073278 20943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:26:42.074518 20943 hybrid_clock.cc:648] HybridClock initialized: now 1714660002074443 us; error 69 us; skew 500 ppm
I20240502 14:26:42.074891 20943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:26:42.077345 20943 webserver.cc:460] Webserver started at http://127.5.107.72:36821/ using document root <none> and password file <none>
I20240502 14:26:42.078099 20943 fs_manager.cc:359] Metadata directory not provided
I20240502 14:26:42.078220 20943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:26:42.084067 20943 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20240502 14:26:42.087313 20960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:42.088797 20943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 14:26:42.088985 20943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "ff6acba046404ef0a6f41ad5e09d852f"
format_stamp: "Formatted at 2024-05-02 14:26:23 on dist-test-slave-2cw1"
I20240502 14:26:42.089897 20943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:26:42.108350 20943 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:26:42.109318 20943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:26:42.109575 20943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:26:42.110519 20943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:26:42.112097 20943 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:26:42.112179 20943 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:42.112309 20943 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:26:42.112365 20943 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:26:42.154469 20943 rpc_server.cc:307] RPC server started. Bound to: 127.5.107.72:35871
I20240502 14:26:42.154522 21075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.107.72:35871 every 8 connection(s)
I20240502 14:26:42.155993 20943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 14:26:42.160743 20806 heartbeater.cc:503] Master 127.5.107.126:33551 was elected leader, sending a full tablet report...
I20240502 14:26:42.164420 5549 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu as pid 20943
I20240502 14:26:42.173841 21076 heartbeater.cc:347] Connected to a master server at 127.5.107.126:33551
I20240502 14:26:42.174213 21076 heartbeater.cc:464] Registering TS with master...
I20240502 14:26:42.175037 21076 heartbeater.cc:511] Master 127.5.107.126:33551 requested a full tablet report, sending...
I20240502 14:26:42.176416 18551 ts_manager.cc:194] Re-registered known tserver with Master: ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871)
I20240502 14:26:42.177172 18551 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.107.72:45493
I20240502 14:26:42.675864 20939 heartbeater.cc:503] Master 127.5.107.126:33551 was elected leader, sending a full tablet report...
I20240502 14:26:43.179675 21076 heartbeater.cc:503] Master 127.5.107.126:33551 was elected leader, sending a full tablet report...
I20240502 14:26:43.303647 19076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:26:43.318241 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:26:43.370654 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:43.398237 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:43.430502 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:43.476456 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:43.493427 21007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:43.498883 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:26:43.904233 19076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:26:43.922276 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:26:43.958587 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:44.043359 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:44.063849 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:44.079193 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:44.098162 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:26:44.200065 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:44.228721 19076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:26:44.228974 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:26:44.230469 21007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:44.245461 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:44.257977 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:26:44.290889 19076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:26:44.329471 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:44.352962 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:44.361949 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:44.376098 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:44.406035 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:44.429783 21007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:44.433231 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:26:44.499168 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:26:44.515728 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:44.541643 21007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:44.919075 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:26:44.919626 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:26:44.942179 19076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:26:44.988842 21007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:45.018527 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:45.029121 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:45.057425 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:45.057971 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:45.251958 19076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:26:45.267587 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:26:45.300908 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:45.340981 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:45.341405 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:45.344239 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:45.370059 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:26:45.371425 21007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:45.839398 19076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:26:45.923060 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:26:45.960503 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:45.979333 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:46.022445 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:46.072067 19076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:26:46.074215 21007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:46.111501 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:46.112802 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:46.117604 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:26:46.141577 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:46.161872 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:26:46.174737 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:46.184968 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:26:46.201532 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:46.243387 21007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:46.259804 19076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:26:46.269374 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:26:46.290619 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:46.298626 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:46.310325 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:46.317046 21007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:46.322671 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:46.335397 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:26:46.646512 19076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:26:46.655539 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:26:46.662673 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:46.712540 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:46.742764 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 14:26:46.841387 18769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.67:45409, user_credentials={real_user=slave}} blocked reactor thread for 50062us
I20240502 14:26:46.947388 21007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:47.021134 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:26:47.028127 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:47.240593 19076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:26:47.255519 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:47.255776 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:26:47.259588 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:47.261384 21007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:47.263165 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:47.290987 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:26:47.326592 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:47.827986 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:26:47.886412 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:47.892642 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:47.949209 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:47.951269 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:47.996085 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:26:48.013507 21007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:48.050252 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:48.051565 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:48.051951 19076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 14:26:48.058876 18637 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.68:41463, user_credentials={real_user=slave}} blocked reactor thread for 66703.8us
I20240502 14:26:48.104440 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:26:48.107370 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:26:48.121721 19076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:26:48.131403 21007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:48.158681 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:48.194590 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:26:48.205597 19029 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.66:33009, user_credentials={real_user=slave}} blocked reactor thread for 166909us
I20240502 14:26:48.242242 18833 consensus_queue.cc:237] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:48.257805 18701 raft_consensus.cc:1261] T 9580f3fdf7084cc0923b1b16bfedd450 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:48.259269 18963 raft_consensus.cc:1261] T 9580f3fdf7084cc0923b1b16bfedd450 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:48.259533 21615 consensus_queue.cc:1035] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:48.261046 21718 consensus_queue.cc:1035] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:48.287134 19076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:26:48.288744 21718 raft_consensus.cc:2932] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:48.289747 18701 raft_consensus.cc:2932] T 9580f3fdf7084cc0923b1b16bfedd450 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:48.298117 18558 catalog_manager.cc:5484] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:48.300238 18963 raft_consensus.cc:2932] T 9580f3fdf7084cc0923b1b16bfedd450 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
W20240502 14:26:48.352013 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9580f3fdf7084cc0923b1b16bfedd450. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:48.365720 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:48.367182 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:26:48.402040 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:48.421701 21007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:48.435911 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:48.440527 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:48.492285 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:26:48.667624 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:26:48.674953 19076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:26:48.736647 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:48.743350 21861 ts_tablet_manager.cc:902] T 9580f3fdf7084cc0923b1b16bfedd450 P a834edffbcb2469aae598b069aab0681: Initiating tablet copy from peer fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
I20240502 14:26:48.763450 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:48.772094 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:48.822023 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:48.834161 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:26:48.835491 21007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:48.867029 21861 tablet_copy_client.cc:308] T 9580f3fdf7084cc0923b1b16bfedd450 P a834edffbcb2469aae598b069aab0681: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:33009
I20240502 14:26:48.905464 18854 tablet_copy_service.cc:140] P fec04c50dc5b49878ffa3c3873df32e4: Received BeginTabletCopySession request for tablet 9580f3fdf7084cc0923b1b16bfedd450 from peer a834edffbcb2469aae598b069aab0681 ({username='slave'} at 127.5.107.69:37387)
I20240502 14:26:48.905763 18854 tablet_copy_service.cc:161] P fec04c50dc5b49878ffa3c3873df32e4: Beginning new tablet copy session on tablet 9580f3fdf7084cc0923b1b16bfedd450 from peer a834edffbcb2469aae598b069aab0681 at {username='slave'} at 127.5.107.69:37387: session id = a834edffbcb2469aae598b069aab0681-9580f3fdf7084cc0923b1b16bfedd450
I20240502 14:26:48.909606 18854 tablet_copy_source_session.cc:205] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:48.916605 21861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9580f3fdf7084cc0923b1b16bfedd450. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:48.946177 21861 tablet_copy_client.cc:783] T 9580f3fdf7084cc0923b1b16bfedd450 P a834edffbcb2469aae598b069aab0681: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:48.946755 21861 tablet_copy_client.cc:647] T 9580f3fdf7084cc0923b1b16bfedd450 P a834edffbcb2469aae598b069aab0681: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:48.950634 21861 tablet_copy_client.cc:523] T 9580f3fdf7084cc0923b1b16bfedd450 P a834edffbcb2469aae598b069aab0681: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:48.956928 21861 tablet_bootstrap.cc:492] T 9580f3fdf7084cc0923b1b16bfedd450 P a834edffbcb2469aae598b069aab0681: Bootstrap starting.
I20240502 14:26:48.988874 18833 consensus_queue.cc:237] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:26:48.992934 18701 raft_consensus.cc:1261] T 9580f3fdf7084cc0923b1b16bfedd450 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:26:48.993405 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:48.993899 21718 consensus_queue.cc:1035] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:49.004778 18701 consensus_queue.cc:237] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:26:49.037776 21861 log.cc:824] T 9580f3fdf7084cc0923b1b16bfedd450 P a834edffbcb2469aae598b069aab0681: Log is configured to *not* fsync() on all Append() calls
I20240502 14:26:49.043982 18963 raft_consensus.cc:1261] T 9580f3fdf7084cc0923b1b16bfedd450 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:49.044783 21718 consensus_queue.cc:1035] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:49.054214 21718 raft_consensus.cc:2932] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:49.061570 18963 raft_consensus.cc:2932] T 9580f3fdf7084cc0923b1b16bfedd450 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
W20240502 14:26:49.069494 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9580f3fdf7084cc0923b1b16bfedd450. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:49.082871 18700 raft_consensus.cc:2932] T 9580f3fdf7084cc0923b1b16bfedd450 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:49.089865 18554 catalog_manager.cc:5484] T 9580f3fdf7084cc0923b1b16bfedd450 P c902805060df4df9b5599d0e94b2dc06 reported cstate change: config changed from index 4 to 5, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:49.120919 18963 raft_consensus.cc:1261] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:49.121650 18833 raft_consensus.cc:1261] T 0ffb6e582ef04fb38dd273c13febac58 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:49.123136 21682 consensus_queue.cc:1035] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:49.125044 21682 consensus_queue.cc:1035] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:49.158186 21682 raft_consensus.cc:2932] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:49.164714 18832 raft_consensus.cc:2932] T 0ffb6e582ef04fb38dd273c13febac58 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
W20240502 14:26:49.171279 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ffb6e582ef04fb38dd273c13febac58. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:49.174444 18963 raft_consensus.cc:2932] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:49.175077 18558 catalog_manager.cc:5484] T 0ffb6e582ef04fb38dd273c13febac58 P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index -1 to 4, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:49.224488 21861 tablet_bootstrap.cc:492] T 9580f3fdf7084cc0923b1b16bfedd450 P a834edffbcb2469aae598b069aab0681: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:49.225436 21861 tablet_bootstrap.cc:492] T 9580f3fdf7084cc0923b1b16bfedd450 P a834edffbcb2469aae598b069aab0681: Bootstrap complete.
I20240502 14:26:49.226208 21861 ts_tablet_manager.cc:1367] T 9580f3fdf7084cc0923b1b16bfedd450 P a834edffbcb2469aae598b069aab0681: Time spent bootstrapping tablet: real 0.270s user 0.079s sys 0.006s
I20240502 14:26:49.235937 21861 raft_consensus.cc:348] T 9580f3fdf7084cc0923b1b16bfedd450 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:49.236804 21861 raft_consensus.cc:729] T 9580f3fdf7084cc0923b1b16bfedd450 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a834edffbcb2469aae598b069aab0681, State: Initialized, Role: FOLLOWER
I20240502 14:26:49.237843 21861 consensus_queue.cc:260] T 9580f3fdf7084cc0923b1b16bfedd450 P a834edffbcb2469aae598b069aab0681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:49.294862 21861 ts_tablet_manager.cc:1398] T 9580f3fdf7084cc0923b1b16bfedd450 P a834edffbcb2469aae598b069aab0681: Time spent starting tablet: real 0.068s user 0.028s sys 0.031s
I20240502 14:26:49.296976 18854 tablet_copy_service.cc:342] P fec04c50dc5b49878ffa3c3873df32e4: Request end of tablet copy session a834edffbcb2469aae598b069aab0681-9580f3fdf7084cc0923b1b16bfedd450 received from {username='slave'} at 127.5.107.69:37387
I20240502 14:26:49.297152 18854 tablet_copy_service.cc:433] P fec04c50dc5b49878ffa3c3873df32e4: ending tablet copy session a834edffbcb2469aae598b069aab0681-9580f3fdf7084cc0923b1b16bfedd450 on tablet 9580f3fdf7084cc0923b1b16bfedd450 with peer a834edffbcb2469aae598b069aab0681
I20240502 14:26:49.305475 19093 consensus_queue.cc:237] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:26:49.309661 18963 raft_consensus.cc:1261] T 332e4b23751b422ab7bd17b80c93701d P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:49.310425 21520 consensus_queue.cc:1035] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:49.314337 18832 raft_consensus.cc:1261] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:49.315119 21520 consensus_queue.cc:1035] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:49.332805 18833 consensus_queue.cc:237] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
W20240502 14:26:49.337709 19031 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 332e4b23751b422ab7bd17b80c93701d. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:49.341451 21520 raft_consensus.cc:2932] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:49.349407 18832 raft_consensus.cc:2932] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:49.359505 18963 raft_consensus.cc:1261] T 6acde0e5509340fbb52bca18645a77b7 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:49.362403 18554 catalog_manager.cc:5484] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index -1 to 4, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:49.370010 19097 raft_consensus.cc:1261] T 6acde0e5509340fbb52bca18645a77b7 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:49.370185 21904 consensus_queue.cc:1035] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:49.375080 21904 consensus_queue.cc:1035] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:49.385797 18963 raft_consensus.cc:2932] T 332e4b23751b422ab7bd17b80c93701d P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:49.404815 21904 raft_consensus.cc:2932] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
W20240502 14:26:49.407271 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6acde0e5509340fbb52bca18645a77b7. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:49.409047 18965 raft_consensus.cc:2932] T 6acde0e5509340fbb52bca18645a77b7 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:49.417888 18554 catalog_manager.cc:5484] T 6acde0e5509340fbb52bca18645a77b7 P c902805060df4df9b5599d0e94b2dc06 reported cstate change: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:49.423031 19093 raft_consensus.cc:2932] T 6acde0e5509340fbb52bca18645a77b7 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:49.448297 19093 consensus_queue.cc:237] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:26:49.453279 18701 raft_consensus.cc:1261] T e5fb3b049ba94af998791640bd0e31c5 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:49.453294 18833 raft_consensus.cc:1261] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:49.454017 21520 consensus_queue.cc:1035] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:49.459143 21520 consensus_queue.cc:1035] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:49.471324 21520 raft_consensus.cc:2932] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:49.479691 18833 raft_consensus.cc:2932] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
W20240502 14:26:49.484175 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5fb3b049ba94af998791640bd0e31c5. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:49.492519 19093 consensus_queue.cc:237] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:26:49.492964 18701 raft_consensus.cc:2932] T e5fb3b049ba94af998791640bd0e31c5 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
W20240502 14:26:49.499554 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8937c809610542d88fdde3daf002c890. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:49.500900 18832 raft_consensus.cc:1261] T 8937c809610542d88fdde3daf002c890 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:49.502550 21949 consensus_queue.cc:1035] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:49.504729 18965 raft_consensus.cc:1261] T 8937c809610542d88fdde3daf002c890 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:49.505466 21520 consensus_queue.cc:1035] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:49.505548 18554 catalog_manager.cc:5484] T e5fb3b049ba94af998791640bd0e31c5 P 2db304b538e14857b7b1cc272d0db94b reported cstate change: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:49.518684 22007 ts_tablet_manager.cc:902] T 9580f3fdf7084cc0923b1b16bfedd450 P ff6acba046404ef0a6f41ad5e09d852f: Initiating tablet copy from peer fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
I20240502 14:26:49.522150 21968 raft_consensus.cc:2932] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:49.525836 18833 raft_consensus.cc:2932] T 8937c809610542d88fdde3daf002c890 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:49.532897 18965 raft_consensus.cc:2932] T 8937c809610542d88fdde3daf002c890 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:49.534725 18558 catalog_manager.cc:5484] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: config changed from index -1 to 4, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:49.547394 20627 raft_consensus.cc:2932] T 9580f3fdf7084cc0923b1b16bfedd450 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:49.574784 22007 tablet_copy_client.cc:308] T 9580f3fdf7084cc0923b1b16bfedd450 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:33009
I20240502 14:26:49.612290 18854 tablet_copy_service.cc:140] P fec04c50dc5b49878ffa3c3873df32e4: Received BeginTabletCopySession request for tablet 9580f3fdf7084cc0923b1b16bfedd450 from peer ff6acba046404ef0a6f41ad5e09d852f ({username='slave'} at 127.5.107.72:58697)
I20240502 14:26:49.612603 18854 tablet_copy_service.cc:161] P fec04c50dc5b49878ffa3c3873df32e4: Beginning new tablet copy session on tablet 9580f3fdf7084cc0923b1b16bfedd450 from peer ff6acba046404ef0a6f41ad5e09d852f at {username='slave'} at 127.5.107.72:58697: session id = ff6acba046404ef0a6f41ad5e09d852f-9580f3fdf7084cc0923b1b16bfedd450
I20240502 14:26:49.616111 18854 tablet_copy_source_session.cc:205] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:49.623515 22007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9580f3fdf7084cc0923b1b16bfedd450. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:49.624358 18832 consensus_queue.cc:237] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:49.628543 18701 raft_consensus.cc:1261] T 9580f3fdf7084cc0923b1b16bfedd450 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:26:49.631960 18965 raft_consensus.cc:1261] T 9580f3fdf7084cc0923b1b16bfedd450 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:26:49.632828 21904 consensus_queue.cc:1035] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:26:49.634047 21904 consensus_queue.cc:1035] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 14:26:49.634346 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9580f3fdf7084cc0923b1b16bfedd450. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:49.642339 20627 raft_consensus.cc:1261] T 9580f3fdf7084cc0923b1b16bfedd450 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:26:49.643065 21904 consensus_queue.cc:1035] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:26:49.648126 20626 raft_consensus.cc:2932] T 9580f3fdf7084cc0923b1b16bfedd450 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:49.646515 21904 raft_consensus.cc:2932] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:49.655488 18558 catalog_manager.cc:5484] T 9580f3fdf7084cc0923b1b16bfedd450 P a834edffbcb2469aae598b069aab0681 reported cstate change: config changed from index 5 to 6, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:49.663823 18701 raft_consensus.cc:2932] T 9580f3fdf7084cc0923b1b16bfedd450 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:49.670162 18965 raft_consensus.cc:2932] T 9580f3fdf7084cc0923b1b16bfedd450 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
W20240502 14:26:49.702047 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9580f3fdf7084cc0923b1b16bfedd450. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:49.702333 22007 tablet_copy_client.cc:783] T 9580f3fdf7084cc0923b1b16bfedd450 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:49.702775 22007 tablet_copy_client.cc:647] T 9580f3fdf7084cc0923b1b16bfedd450 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:49.706943 22007 tablet_copy_client.cc:523] T 9580f3fdf7084cc0923b1b16bfedd450 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:49.719998 22007 tablet_bootstrap.cc:492] T 9580f3fdf7084cc0923b1b16bfedd450 P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap starting.
I20240502 14:26:49.724716 22073 ts_tablet_manager.cc:902] T 0ffb6e582ef04fb38dd273c13febac58 P ff6acba046404ef0a6f41ad5e09d852f: Initiating tablet copy from peer 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:26:49.742713 18965 consensus_queue.cc:237] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:49.750855 18701 raft_consensus.cc:1261] T cf2a6426ae6c438f942d39530da6e3e3 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:49.751677 21881 consensus_queue.cc:1035] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:49.754000 22007 log.cc:824] T 9580f3fdf7084cc0923b1b16bfedd450 P ff6acba046404ef0a6f41ad5e09d852f: Log is configured to *not* fsync() on all Append() calls
I20240502 14:26:49.775328 22095 ts_tablet_manager.cc:902] T 332e4b23751b422ab7bd17b80c93701d P ff6acba046404ef0a6f41ad5e09d852f: Initiating tablet copy from peer 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:26:49.798667 22095 tablet_copy_client.cc:308] T 332e4b23751b422ab7bd17b80c93701d P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.68:41463
I20240502 14:26:49.805119 19093 raft_consensus.cc:1261] T cf2a6426ae6c438f942d39530da6e3e3 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:49.805866 21881 consensus_queue.cc:1035] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:49.807507 19117 tablet_copy_service.cc:140] P 007b2e79dc424a6f80141180f072f2d8: Received BeginTabletCopySession request for tablet 332e4b23751b422ab7bd17b80c93701d from peer ff6acba046404ef0a6f41ad5e09d852f ({username='slave'} at 127.5.107.72:33829)
I20240502 14:26:49.807776 19117 tablet_copy_service.cc:161] P 007b2e79dc424a6f80141180f072f2d8: Beginning new tablet copy session on tablet 332e4b23751b422ab7bd17b80c93701d from peer ff6acba046404ef0a6f41ad5e09d852f at {username='slave'} at 127.5.107.72:33829: session id = ff6acba046404ef0a6f41ad5e09d852f-332e4b23751b422ab7bd17b80c93701d
I20240502 14:26:49.813836 19117 tablet_copy_source_session.cc:205] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:49.816877 22095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 332e4b23751b422ab7bd17b80c93701d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 14:26:49.822026 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf2a6426ae6c438f942d39530da6e3e3. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:49.837700 22095 tablet_copy_client.cc:783] T 332e4b23751b422ab7bd17b80c93701d P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:49.838120 22095 tablet_copy_client.cc:647] T 332e4b23751b422ab7bd17b80c93701d P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:49.843978 22095 tablet_copy_client.cc:523] T 332e4b23751b422ab7bd17b80c93701d P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:49.855130 22114 raft_consensus.cc:2932] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:49.869074 22095 tablet_bootstrap.cc:492] T 332e4b23751b422ab7bd17b80c93701d P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap starting.
I20240502 14:26:49.873960 19093 raft_consensus.cc:2932] T cf2a6426ae6c438f942d39530da6e3e3 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:49.877719 18701 raft_consensus.cc:2932] T cf2a6426ae6c438f942d39530da6e3e3 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:49.879869 18832 consensus_queue.cc:237] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:49.884400 18557 catalog_manager.cc:5484] T cf2a6426ae6c438f942d39530da6e3e3 P 2db304b538e14857b7b1cc272d0db94b reported cstate change: config changed from index -1 to 4, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:49.926733 22126 ts_tablet_manager.cc:902] T e5fb3b049ba94af998791640bd0e31c5 P a14c220ad2cc4bf9bd7dad65f4803eb0: Initiating tablet copy from peer 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
W20240502 14:26:49.934971 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d8667d91fd64809b635362c4a8fa33f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:49.947074 18701 raft_consensus.cc:1261] T 0d8667d91fd64809b635362c4a8fa33f P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:49.947963 21904 consensus_queue.cc:1035] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:49.958086 22133 ts_tablet_manager.cc:902] T 8937c809610542d88fdde3daf002c890 P ff6acba046404ef0a6f41ad5e09d852f: Initiating tablet copy from peer 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:26:49.958272 19097 raft_consensus.cc:1261] T 0d8667d91fd64809b635362c4a8fa33f P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:49.959081 21904 consensus_queue.cc:1035] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:49.978348 21892 raft_consensus.cc:2932] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:49.979293 22007 tablet_bootstrap.cc:492] T 9580f3fdf7084cc0923b1b16bfedd450 P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:49.979962 22007 tablet_bootstrap.cc:492] T 9580f3fdf7084cc0923b1b16bfedd450 P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap complete.
I20240502 14:26:49.980512 22007 ts_tablet_manager.cc:1367] T 9580f3fdf7084cc0923b1b16bfedd450 P ff6acba046404ef0a6f41ad5e09d852f: Time spent bootstrapping tablet: real 0.261s user 0.081s sys 0.000s
I20240502 14:26:49.982214 18700 raft_consensus.cc:2932] T 0d8667d91fd64809b635362c4a8fa33f P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:49.984784 19097 raft_consensus.cc:2932] T 0d8667d91fd64809b635362c4a8fa33f P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:49.987679 18557 catalog_manager.cc:5484] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index -1 to 4, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:49.997249 22073 tablet_copy_client.cc:308] T 0ffb6e582ef04fb38dd273c13febac58 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.65:36749
I20240502 14:26:50.007673 22137 ts_tablet_manager.cc:902] T 6acde0e5509340fbb52bca18645a77b7 P a14c220ad2cc4bf9bd7dad65f4803eb0: Initiating tablet copy from peer fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
I20240502 14:26:50.026767 22126 tablet_copy_client.cc:308] T e5fb3b049ba94af998791640bd0e31c5 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.68:41463
I20240502 14:26:50.039893 22007 raft_consensus.cc:348] T 9580f3fdf7084cc0923b1b16bfedd450 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:26:50.040472 22007 raft_consensus.cc:729] T 9580f3fdf7084cc0923b1b16bfedd450 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6acba046404ef0a6f41ad5e09d852f, State: Initialized, Role: FOLLOWER
I20240502 14:26:50.057560 22133 tablet_copy_client.cc:308] T 8937c809610542d88fdde3daf002c890 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.68:41463
I20240502 14:26:50.058140 18721 tablet_copy_service.cc:140] P 2db304b538e14857b7b1cc272d0db94b: Received BeginTabletCopySession request for tablet 0ffb6e582ef04fb38dd273c13febac58 from peer ff6acba046404ef0a6f41ad5e09d852f ({username='slave'} at 127.5.107.72:37745)
I20240502 14:26:50.058431 18721 tablet_copy_service.cc:161] P 2db304b538e14857b7b1cc272d0db94b: Beginning new tablet copy session on tablet 0ffb6e582ef04fb38dd273c13febac58 from peer ff6acba046404ef0a6f41ad5e09d852f at {username='slave'} at 127.5.107.72:37745: session id = ff6acba046404ef0a6f41ad5e09d852f-0ffb6e582ef04fb38dd273c13febac58
I20240502 14:26:50.062546 18721 tablet_copy_source_session.cc:205] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:50.064966 22073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ffb6e582ef04fb38dd273c13febac58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:50.065327 22007 consensus_queue.cc:260] T 9580f3fdf7084cc0923b1b16bfedd450 P ff6acba046404ef0a6f41ad5e09d852f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:26:50.080425 19117 tablet_copy_service.cc:140] P 007b2e79dc424a6f80141180f072f2d8: Received BeginTabletCopySession request for tablet e5fb3b049ba94af998791640bd0e31c5 from peer a14c220ad2cc4bf9bd7dad65f4803eb0 ({username='slave'} at 127.5.107.70:47193)
I20240502 14:26:50.080693 19117 tablet_copy_service.cc:161] P 007b2e79dc424a6f80141180f072f2d8: Beginning new tablet copy session on tablet e5fb3b049ba94af998791640bd0e31c5 from peer a14c220ad2cc4bf9bd7dad65f4803eb0 at {username='slave'} at 127.5.107.70:47193: session id = a14c220ad2cc4bf9bd7dad65f4803eb0-e5fb3b049ba94af998791640bd0e31c5
I20240502 14:26:50.084065 19117 tablet_copy_source_session.cc:205] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:50.095896 22126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5fb3b049ba94af998791640bd0e31c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:50.105404 22073 tablet_copy_client.cc:783] T 0ffb6e582ef04fb38dd273c13febac58 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:50.106086 22073 tablet_copy_client.cc:647] T 0ffb6e582ef04fb38dd273c13febac58 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:50.118831 22137 tablet_copy_client.cc:308] T 6acde0e5509340fbb52bca18645a77b7 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:33009
I20240502 14:26:50.163606 22073 tablet_copy_client.cc:523] T 0ffb6e582ef04fb38dd273c13febac58 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:50.164424 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:50.168543 22073 tablet_bootstrap.cc:492] T 0ffb6e582ef04fb38dd273c13febac58 P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap starting.
I20240502 14:26:50.190860 22007 ts_tablet_manager.cc:1398] T 9580f3fdf7084cc0923b1b16bfedd450 P ff6acba046404ef0a6f41ad5e09d852f: Time spent starting tablet: real 0.210s user 0.017s sys 0.035s
I20240502 14:26:50.191854 18854 tablet_copy_service.cc:342] P fec04c50dc5b49878ffa3c3873df32e4: Request end of tablet copy session ff6acba046404ef0a6f41ad5e09d852f-9580f3fdf7084cc0923b1b16bfedd450 received from {username='slave'} at 127.5.107.72:58697
I20240502 14:26:50.192016 18854 tablet_copy_service.cc:433] P fec04c50dc5b49878ffa3c3873df32e4: ending tablet copy session ff6acba046404ef0a6f41ad5e09d852f-9580f3fdf7084cc0923b1b16bfedd450 on tablet 9580f3fdf7084cc0923b1b16bfedd450 with peer ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:26:50.225347 22184 ts_tablet_manager.cc:902] T 9580f3fdf7084cc0923b1b16bfedd450 P 7ab6d395b30d4afb9821b61c9454430b: Initiating tablet copy from peer fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
I20240502 14:26:50.226085 19117 tablet_copy_service.cc:140] P 007b2e79dc424a6f80141180f072f2d8: Received BeginTabletCopySession request for tablet 8937c809610542d88fdde3daf002c890 from peer ff6acba046404ef0a6f41ad5e09d852f ({username='slave'} at 127.5.107.72:33829)
I20240502 14:26:50.226330 19117 tablet_copy_service.cc:161] P 007b2e79dc424a6f80141180f072f2d8: Beginning new tablet copy session on tablet 8937c809610542d88fdde3daf002c890 from peer ff6acba046404ef0a6f41ad5e09d852f at {username='slave'} at 127.5.107.72:33829: session id = ff6acba046404ef0a6f41ad5e09d852f-8937c809610542d88fdde3daf002c890
I20240502 14:26:50.232128 19117 tablet_copy_source_session.cc:205] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:50.239225 22095 tablet_bootstrap.cc:492] T 332e4b23751b422ab7bd17b80c93701d P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:50.239799 22095 tablet_bootstrap.cc:492] T 332e4b23751b422ab7bd17b80c93701d P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap complete.
I20240502 14:26:50.240273 22095 ts_tablet_manager.cc:1367] T 332e4b23751b422ab7bd17b80c93701d P ff6acba046404ef0a6f41ad5e09d852f: Time spent bootstrapping tablet: real 0.371s user 0.062s sys 0.027s
I20240502 14:26:50.241650 22095 raft_consensus.cc:348] T 332e4b23751b422ab7bd17b80c93701d P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:26:50.242107 22095 raft_consensus.cc:729] T 332e4b23751b422ab7bd17b80c93701d P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6acba046404ef0a6f41ad5e09d852f, State: Initialized, Role: FOLLOWER
I20240502 14:26:50.242470 22095 consensus_queue.cc:260] T 332e4b23751b422ab7bd17b80c93701d P ff6acba046404ef0a6f41ad5e09d852f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:26:50.243362 22095 ts_tablet_manager.cc:1398] T 332e4b23751b422ab7bd17b80c93701d P ff6acba046404ef0a6f41ad5e09d852f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:26:50.252207 22133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8937c809610542d88fdde3daf002c890. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:50.267968 19117 tablet_copy_service.cc:342] P 007b2e79dc424a6f80141180f072f2d8: Request end of tablet copy session ff6acba046404ef0a6f41ad5e09d852f-332e4b23751b422ab7bd17b80c93701d received from {username='slave'} at 127.5.107.72:33829
I20240502 14:26:50.268245 19117 tablet_copy_service.cc:433] P 007b2e79dc424a6f80141180f072f2d8: ending tablet copy session ff6acba046404ef0a6f41ad5e09d852f-332e4b23751b422ab7bd17b80c93701d on tablet 332e4b23751b422ab7bd17b80c93701d with peer ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:26:50.270799 22133 tablet_copy_client.cc:783] T 8937c809610542d88fdde3daf002c890 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:50.271529 22133 tablet_copy_client.cc:647] T 8937c809610542d88fdde3daf002c890 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:50.283401 22200 ts_tablet_manager.cc:902] T cf2a6426ae6c438f942d39530da6e3e3 P 7ab6d395b30d4afb9821b61c9454430b: Initiating tablet copy from peer c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:26:50.285499 18854 tablet_copy_service.cc:140] P fec04c50dc5b49878ffa3c3873df32e4: Received BeginTabletCopySession request for tablet 6acde0e5509340fbb52bca18645a77b7 from peer a14c220ad2cc4bf9bd7dad65f4803eb0 ({username='slave'} at 127.5.107.70:57723)
I20240502 14:26:50.285723 18854 tablet_copy_service.cc:161] P fec04c50dc5b49878ffa3c3873df32e4: Beginning new tablet copy session on tablet 6acde0e5509340fbb52bca18645a77b7 from peer a14c220ad2cc4bf9bd7dad65f4803eb0 at {username='slave'} at 127.5.107.70:57723: session id = a14c220ad2cc4bf9bd7dad65f4803eb0-6acde0e5509340fbb52bca18645a77b7
I20240502 14:26:50.287312 22133 tablet_copy_client.cc:523] T 8937c809610542d88fdde3daf002c890 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:50.288950 18854 tablet_copy_source_session.cc:205] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:50.292461 22133 tablet_bootstrap.cc:492] T 8937c809610542d88fdde3daf002c890 P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap starting.
I20240502 14:26:50.296700 22184 tablet_copy_client.cc:308] T 9580f3fdf7084cc0923b1b16bfedd450 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:33009
I20240502 14:26:50.307071 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:50.348892 22137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6acde0e5509340fbb52bca18645a77b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:50.364064 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:26:50.389709 21007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:50.483783 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:26:50.503340 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:50.528035 19076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:26:50.556593 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:50.568758 22207 ts_tablet_manager.cc:902] T 0d8667d91fd64809b635362c4a8fa33f P 7ab6d395b30d4afb9821b61c9454430b: Initiating tablet copy from peer fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
I20240502 14:26:50.582351 19093 consensus_queue.cc:237] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:50.590483 18854 tablet_copy_service.cc:140] P fec04c50dc5b49878ffa3c3873df32e4: Received BeginTabletCopySession request for tablet 9580f3fdf7084cc0923b1b16bfedd450 from peer 7ab6d395b30d4afb9821b61c9454430b ({username='slave'} at 127.5.107.71:44135)
I20240502 14:26:50.590724 18854 tablet_copy_service.cc:161] P fec04c50dc5b49878ffa3c3873df32e4: Beginning new tablet copy session on tablet 9580f3fdf7084cc0923b1b16bfedd450 from peer 7ab6d395b30d4afb9821b61c9454430b at {username='slave'} at 127.5.107.71:44135: session id = 7ab6d395b30d4afb9821b61c9454430b-9580f3fdf7084cc0923b1b16bfedd450
I20240502 14:26:50.594877 18854 tablet_copy_source_session.cc:205] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:50.602715 22184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9580f3fdf7084cc0923b1b16bfedd450. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 14:26:50.614797 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:50.616747 18832 raft_consensus.cc:1261] T 8937c809610542d88fdde3daf002c890 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:50.618306 21968 consensus_queue.cc:1035] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:50.621305 18965 raft_consensus.cc:1261] T 8937c809610542d88fdde3daf002c890 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:50.622128 21968 consensus_queue.cc:1035] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:50.642067 22184 tablet_copy_client.cc:783] T 9580f3fdf7084cc0923b1b16bfedd450 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:50.642477 22184 tablet_copy_client.cc:647] T 9580f3fdf7084cc0923b1b16bfedd450 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:50.646445 22184 tablet_copy_client.cc:523] T 9580f3fdf7084cc0923b1b16bfedd450 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:50.655921 22137 tablet_copy_client.cc:783] T 6acde0e5509340fbb52bca18645a77b7 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:50.659260 22184 tablet_bootstrap.cc:492] T 9580f3fdf7084cc0923b1b16bfedd450 P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap starting.
W20240502 14:26:50.664043 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8937c809610542d88fdde3daf002c890. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:50.666880 22137 tablet_copy_client.cc:647] T 6acde0e5509340fbb52bca18645a77b7 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:50.670199 22200 tablet_copy_client.cc:308] T cf2a6426ae6c438f942d39530da6e3e3 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.67:45409
I20240502 14:26:50.672470 22207 tablet_copy_client.cc:308] T 0d8667d91fd64809b635362c4a8fa33f P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:33009
I20240502 14:26:50.673408 18854 tablet_copy_service.cc:140] P fec04c50dc5b49878ffa3c3873df32e4: Received BeginTabletCopySession request for tablet 0d8667d91fd64809b635362c4a8fa33f from peer 7ab6d395b30d4afb9821b61c9454430b ({username='slave'} at 127.5.107.71:44135)
I20240502 14:26:50.673591 18854 tablet_copy_service.cc:161] P fec04c50dc5b49878ffa3c3873df32e4: Beginning new tablet copy session on tablet 0d8667d91fd64809b635362c4a8fa33f from peer 7ab6d395b30d4afb9821b61c9454430b at {username='slave'} at 127.5.107.71:44135: session id = 7ab6d395b30d4afb9821b61c9454430b-0d8667d91fd64809b635362c4a8fa33f
I20240502 14:26:50.677102 18854 tablet_copy_source_session.cc:205] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:50.678647 22207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d8667d91fd64809b635362c4a8fa33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:50.678898 22137 tablet_copy_client.cc:523] T 6acde0e5509340fbb52bca18645a77b7 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:50.683408 22184 log.cc:824] T 9580f3fdf7084cc0923b1b16bfedd450 P 7ab6d395b30d4afb9821b61c9454430b: Log is configured to *not* fsync() on all Append() calls
I20240502 14:26:50.689322 21029 raft_consensus.cc:2932] T 9580f3fdf7084cc0923b1b16bfedd450 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:50.690025 21949 raft_consensus.cc:2932] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:50.703132 18963 raft_consensus.cc:2932] T 8937c809610542d88fdde3daf002c890 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:50.710506 22137 tablet_bootstrap.cc:492] T 6acde0e5509340fbb52bca18645a77b7 P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap starting.
I20240502 14:26:50.714849 18558 catalog_manager.cc:5484] T 8937c809610542d88fdde3daf002c890 P c902805060df4df9b5599d0e94b2dc06 reported cstate change: config changed from index 4 to 5, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:50.718144 22126 tablet_copy_client.cc:783] T e5fb3b049ba94af998791640bd0e31c5 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:50.722873 22126 tablet_copy_client.cc:647] T e5fb3b049ba94af998791640bd0e31c5 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:50.728689 22126 tablet_copy_client.cc:523] T e5fb3b049ba94af998791640bd0e31c5 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:50.733425 18701 consensus_queue.cc:237] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:26:50.743606 22184 tablet_bootstrap.cc:492] T 9580f3fdf7084cc0923b1b16bfedd450 P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:50.744446 22184 tablet_bootstrap.cc:492] T 9580f3fdf7084cc0923b1b16bfedd450 P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap complete.
I20240502 14:26:50.745019 22184 ts_tablet_manager.cc:1367] T 9580f3fdf7084cc0923b1b16bfedd450 P 7ab6d395b30d4afb9821b61c9454430b: Time spent bootstrapping tablet: real 0.086s user 0.064s sys 0.009s
W20240502 14:26:50.745537 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b315327141954b53a0f87c2d1dc14b9b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:50.746870 19097 raft_consensus.cc:1261] T b315327141954b53a0f87c2d1dc14b9b P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:50.747459 21791 consensus_queue.cc:1035] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:50.752388 22184 raft_consensus.cc:348] T 9580f3fdf7084cc0923b1b16bfedd450 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:50.752938 22184 raft_consensus.cc:729] T 9580f3fdf7084cc0923b1b16bfedd450 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab6d395b30d4afb9821b61c9454430b, State: Initialized, Role: FOLLOWER
I20240502 14:26:50.753832 22184 consensus_queue.cc:260] T 9580f3fdf7084cc0923b1b16bfedd450 P 7ab6d395b30d4afb9821b61c9454430b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:50.762621 22184 ts_tablet_manager.cc:1398] T 9580f3fdf7084cc0923b1b16bfedd450 P 7ab6d395b30d4afb9821b61c9454430b: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20240502 14:26:50.764904 18854 tablet_copy_service.cc:342] P fec04c50dc5b49878ffa3c3873df32e4: Request end of tablet copy session 7ab6d395b30d4afb9821b61c9454430b-9580f3fdf7084cc0923b1b16bfedd450 received from {username='slave'} at 127.5.107.71:44135
I20240502 14:26:50.765103 18854 tablet_copy_service.cc:433] P fec04c50dc5b49878ffa3c3873df32e4: ending tablet copy session 7ab6d395b30d4afb9821b61c9454430b-9580f3fdf7084cc0923b1b16bfedd450 on tablet 9580f3fdf7084cc0923b1b16bfedd450 with peer 7ab6d395b30d4afb9821b61c9454430b
I20240502 14:26:50.767071 18833 raft_consensus.cc:1261] T b315327141954b53a0f87c2d1dc14b9b P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:50.771368 21791 consensus_queue.cc:1035] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:50.775769 22207 tablet_copy_client.cc:783] T 0d8667d91fd64809b635362c4a8fa33f P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:50.776147 22207 tablet_copy_client.cc:647] T 0d8667d91fd64809b635362c4a8fa33f P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:50.780946 22207 tablet_copy_client.cc:523] T 0d8667d91fd64809b635362c4a8fa33f P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:50.791082 22137 log.cc:824] T 6acde0e5509340fbb52bca18645a77b7 P a14c220ad2cc4bf9bd7dad65f4803eb0: Log is configured to *not* fsync() on all Append() calls
I20240502 14:26:50.792586 22207 tablet_bootstrap.cc:492] T 0d8667d91fd64809b635362c4a8fa33f P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap starting.
I20240502 14:26:50.806778 18985 tablet_copy_service.cc:140] P c902805060df4df9b5599d0e94b2dc06: Received BeginTabletCopySession request for tablet cf2a6426ae6c438f942d39530da6e3e3 from peer 7ab6d395b30d4afb9821b61c9454430b ({username='slave'} at 127.5.107.71:55147)
I20240502 14:26:50.807044 18985 tablet_copy_service.cc:161] P c902805060df4df9b5599d0e94b2dc06: Beginning new tablet copy session on tablet cf2a6426ae6c438f942d39530da6e3e3 from peer 7ab6d395b30d4afb9821b61c9454430b at {username='slave'} at 127.5.107.71:55147: session id = 7ab6d395b30d4afb9821b61c9454430b-cf2a6426ae6c438f942d39530da6e3e3
I20240502 14:26:50.807111 18832 raft_consensus.cc:2932] T b315327141954b53a0f87c2d1dc14b9b P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:50.806455 22247 raft_consensus.cc:2932] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:50.810377 21028 raft_consensus.cc:1203] T 332e4b23751b422ab7bd17b80c93701d P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:26:50.813916 18557 catalog_manager.cc:5484] T b315327141954b53a0f87c2d1dc14b9b P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index -1 to 4, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:50.819326 18985 tablet_copy_source_session.cc:205] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:50.822208 22200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf2a6426ae6c438f942d39530da6e3e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:50.824066 18832 raft_consensus.cc:2932] T 8937c809610542d88fdde3daf002c890 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:50.837206 22200 tablet_copy_client.cc:783] T cf2a6426ae6c438f942d39530da6e3e3 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:50.837512 22200 tablet_copy_client.cc:647] T cf2a6426ae6c438f942d39530da6e3e3 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:50.839733 22200 tablet_copy_client.cc:523] T cf2a6426ae6c438f942d39530da6e3e3 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:50.842381 22126 tablet_bootstrap.cc:492] T e5fb3b049ba94af998791640bd0e31c5 P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap starting.
I20240502 14:26:50.845037 22200 tablet_bootstrap.cc:492] T cf2a6426ae6c438f942d39530da6e3e3 P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap starting.
I20240502 14:26:50.853221 19097 raft_consensus.cc:2932] T b315327141954b53a0f87c2d1dc14b9b P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:50.888878 22133 tablet_bootstrap.cc:492] T 8937c809610542d88fdde3daf002c890 P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:50.890138 22133 tablet_bootstrap.cc:492] T 8937c809610542d88fdde3daf002c890 P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap complete.
I20240502 14:26:50.894631 22133 ts_tablet_manager.cc:1367] T 8937c809610542d88fdde3daf002c890 P ff6acba046404ef0a6f41ad5e09d852f: Time spent bootstrapping tablet: real 0.598s user 0.051s sys 0.009s
I20240502 14:26:50.896286 22133 raft_consensus.cc:348] T 8937c809610542d88fdde3daf002c890 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:26:50.896545 22133 raft_consensus.cc:729] T 8937c809610542d88fdde3daf002c890 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6acba046404ef0a6f41ad5e09d852f, State: Initialized, Role: FOLLOWER
I20240502 14:26:50.897743 22133 consensus_queue.cc:260] T 8937c809610542d88fdde3daf002c890 P ff6acba046404ef0a6f41ad5e09d852f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:26:50.898890 22133 ts_tablet_manager.cc:1398] T 8937c809610542d88fdde3daf002c890 P ff6acba046404ef0a6f41ad5e09d852f: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:26:50.899855 19117 tablet_copy_service.cc:342] P 007b2e79dc424a6f80141180f072f2d8: Request end of tablet copy session ff6acba046404ef0a6f41ad5e09d852f-8937c809610542d88fdde3daf002c890 received from {username='slave'} at 127.5.107.72:33829
I20240502 14:26:50.900024 19117 tablet_copy_service.cc:433] P 007b2e79dc424a6f80141180f072f2d8: ending tablet copy session ff6acba046404ef0a6f41ad5e09d852f-8937c809610542d88fdde3daf002c890 on tablet 8937c809610542d88fdde3daf002c890 with peer ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:26:50.953284 18832 consensus_queue.cc:237] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:26:50.956408 22137 tablet_bootstrap.cc:492] T 6acde0e5509340fbb52bca18645a77b7 P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:50.957062 22137 tablet_bootstrap.cc:492] T 6acde0e5509340fbb52bca18645a77b7 P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap complete.
I20240502 14:26:50.957599 22137 ts_tablet_manager.cc:1367] T 6acde0e5509340fbb52bca18645a77b7 P a14c220ad2cc4bf9bd7dad65f4803eb0: Time spent bootstrapping tablet: real 0.247s user 0.074s sys 0.000s
I20240502 14:26:50.972460 22137 raft_consensus.cc:348] T 6acde0e5509340fbb52bca18645a77b7 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:26:50.973137 22137 raft_consensus.cc:729] T 6acde0e5509340fbb52bca18645a77b7 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a14c220ad2cc4bf9bd7dad65f4803eb0, State: Initialized, Role: FOLLOWER
I20240502 14:26:50.974107 22137 consensus_queue.cc:260] T 6acde0e5509340fbb52bca18645a77b7 P a14c220ad2cc4bf9bd7dad65f4803eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:26:50.983522 22137 ts_tablet_manager.cc:1398] T 6acde0e5509340fbb52bca18645a77b7 P a14c220ad2cc4bf9bd7dad65f4803eb0: Time spent starting tablet: real 0.025s user 0.009s sys 0.008s
W20240502 14:26:50.984014 18768 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 522014370f0a4cc3bd31aa00d6a05c5e. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:50.984330 19093 raft_consensus.cc:1261] T 522014370f0a4cc3bd31aa00d6a05c5e P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:50.984963 21892 consensus_queue.cc:1035] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:50.989519 18854 tablet_copy_service.cc:342] P fec04c50dc5b49878ffa3c3873df32e4: Request end of tablet copy session a14c220ad2cc4bf9bd7dad65f4803eb0-6acde0e5509340fbb52bca18645a77b7 received from {username='slave'} at 127.5.107.70:57723
I20240502 14:26:50.989682 18854 tablet_copy_service.cc:433] P fec04c50dc5b49878ffa3c3873df32e4: ending tablet copy session a14c220ad2cc4bf9bd7dad65f4803eb0-6acde0e5509340fbb52bca18645a77b7 on tablet 6acde0e5509340fbb52bca18645a77b7 with peer a14c220ad2cc4bf9bd7dad65f4803eb0
I20240502 14:26:51.009740 18701 raft_consensus.cc:1261] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:51.010443 21892 consensus_queue.cc:1035] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:51.026185 20760 raft_consensus.cc:1203] T 6acde0e5509340fbb52bca18645a77b7 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:26:51.042116 21892 raft_consensus.cc:2932] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:51.044327 22280 ts_tablet_manager.cc:902] T 8937c809610542d88fdde3daf002c890 P a834edffbcb2469aae598b069aab0681: Initiating tablet copy from peer 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:26:51.046224 19097 raft_consensus.cc:2932] T 522014370f0a4cc3bd31aa00d6a05c5e P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:51.052583 18558 catalog_manager.cc:5484] T 522014370f0a4cc3bd31aa00d6a05c5e P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: config changed from index -1 to 4, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:51.054625 18700 raft_consensus.cc:2932] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:51.055400 22126 tablet_bootstrap.cc:492] T e5fb3b049ba94af998791640bd0e31c5 P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:51.056095 22126 tablet_bootstrap.cc:492] T e5fb3b049ba94af998791640bd0e31c5 P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap complete.
I20240502 14:26:51.056385 22126 ts_tablet_manager.cc:1367] T e5fb3b049ba94af998791640bd0e31c5 P a14c220ad2cc4bf9bd7dad65f4803eb0: Time spent bootstrapping tablet: real 0.214s user 0.050s sys 0.003s
I20240502 14:26:51.058686 22126 raft_consensus.cc:348] T e5fb3b049ba94af998791640bd0e31c5 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:26:51.058941 22126 raft_consensus.cc:729] T e5fb3b049ba94af998791640bd0e31c5 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a14c220ad2cc4bf9bd7dad65f4803eb0, State: Initialized, Role: FOLLOWER
I20240502 14:26:51.059257 22126 consensus_queue.cc:260] T e5fb3b049ba94af998791640bd0e31c5 P a14c220ad2cc4bf9bd7dad65f4803eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:26:51.067368 22126 ts_tablet_manager.cc:1398] T e5fb3b049ba94af998791640bd0e31c5 P a14c220ad2cc4bf9bd7dad65f4803eb0: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240502 14:26:51.068154 19117 tablet_copy_service.cc:342] P 007b2e79dc424a6f80141180f072f2d8: Request end of tablet copy session a14c220ad2cc4bf9bd7dad65f4803eb0-e5fb3b049ba94af998791640bd0e31c5 received from {username='slave'} at 127.5.107.70:47193
I20240502 14:26:51.068307 19117 tablet_copy_service.cc:433] P 007b2e79dc424a6f80141180f072f2d8: ending tablet copy session a14c220ad2cc4bf9bd7dad65f4803eb0-e5fb3b049ba94af998791640bd0e31c5 on tablet e5fb3b049ba94af998791640bd0e31c5 with peer a14c220ad2cc4bf9bd7dad65f4803eb0
I20240502 14:26:51.082795 22280 tablet_copy_client.cc:308] T 8937c809610542d88fdde3daf002c890 P a834edffbcb2469aae598b069aab0681: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.68:41463
I20240502 14:26:51.111996 22207 tablet_bootstrap.cc:492] T 0d8667d91fd64809b635362c4a8fa33f P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:51.115406 22207 tablet_bootstrap.cc:492] T 0d8667d91fd64809b635362c4a8fa33f P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap complete.
I20240502 14:26:51.115852 22207 ts_tablet_manager.cc:1367] T 0d8667d91fd64809b635362c4a8fa33f P 7ab6d395b30d4afb9821b61c9454430b: Time spent bootstrapping tablet: real 0.323s user 0.056s sys 0.004s
I20240502 14:26:51.117122 22207 raft_consensus.cc:348] T 0d8667d91fd64809b635362c4a8fa33f P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:51.118011 22200 tablet_bootstrap.cc:492] T cf2a6426ae6c438f942d39530da6e3e3 P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:51.118402 22200 tablet_bootstrap.cc:492] T cf2a6426ae6c438f942d39530da6e3e3 P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap complete.
I20240502 14:26:51.118687 22200 ts_tablet_manager.cc:1367] T cf2a6426ae6c438f942d39530da6e3e3 P 7ab6d395b30d4afb9821b61c9454430b: Time spent bootstrapping tablet: real 0.274s user 0.046s sys 0.012s
I20240502 14:26:51.119410 22200 raft_consensus.cc:348] T cf2a6426ae6c438f942d39530da6e3e3 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:51.119635 22200 raft_consensus.cc:729] T cf2a6426ae6c438f942d39530da6e3e3 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab6d395b30d4afb9821b61c9454430b, State: Initialized, Role: FOLLOWER
I20240502 14:26:51.119874 22200 consensus_queue.cc:260] T cf2a6426ae6c438f942d39530da6e3e3 P 7ab6d395b30d4afb9821b61c9454430b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:51.120494 22200 ts_tablet_manager.cc:1398] T cf2a6426ae6c438f942d39530da6e3e3 P 7ab6d395b30d4afb9821b61c9454430b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:26:51.122306 18985 tablet_copy_service.cc:342] P c902805060df4df9b5599d0e94b2dc06: Request end of tablet copy session 7ab6d395b30d4afb9821b61c9454430b-cf2a6426ae6c438f942d39530da6e3e3 received from {username='slave'} at 127.5.107.71:55147
I20240502 14:26:51.122520 18985 tablet_copy_service.cc:433] P c902805060df4df9b5599d0e94b2dc06: ending tablet copy session 7ab6d395b30d4afb9821b61c9454430b-cf2a6426ae6c438f942d39530da6e3e3 on tablet cf2a6426ae6c438f942d39530da6e3e3 with peer 7ab6d395b30d4afb9821b61c9454430b
I20240502 14:26:51.129277 22207 raft_consensus.cc:729] T 0d8667d91fd64809b635362c4a8fa33f P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab6d395b30d4afb9821b61c9454430b, State: Initialized, Role: FOLLOWER
I20240502 14:26:51.129798 19117 tablet_copy_service.cc:140] P 007b2e79dc424a6f80141180f072f2d8: Received BeginTabletCopySession request for tablet 8937c809610542d88fdde3daf002c890 from peer a834edffbcb2469aae598b069aab0681 ({username='slave'} at 127.5.107.69:37525)
I20240502 14:26:51.129987 19117 tablet_copy_service.cc:161] P 007b2e79dc424a6f80141180f072f2d8: Beginning new tablet copy session on tablet 8937c809610542d88fdde3daf002c890 from peer a834edffbcb2469aae598b069aab0681 at {username='slave'} at 127.5.107.69:37525: session id = a834edffbcb2469aae598b069aab0681-8937c809610542d88fdde3daf002c890
I20240502 14:26:51.130255 22207 consensus_queue.cc:260] T 0d8667d91fd64809b635362c4a8fa33f P 7ab6d395b30d4afb9821b61c9454430b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:51.132982 19117 tablet_copy_source_session.cc:205] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:51.136901 22280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8937c809610542d88fdde3daf002c890. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:51.141233 20894 raft_consensus.cc:1203] T 9580f3fdf7084cc0923b1b16bfedd450 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 14:26:51.144299 18700 consensus_queue.cc:237] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
W20240502 14:26:51.154072 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b315327141954b53a0f87c2d1dc14b9b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:51.160662 19093 raft_consensus.cc:1261] T b315327141954b53a0f87c2d1dc14b9b P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:51.162039 22247 consensus_queue.cc:1035] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:51.174656 22207 ts_tablet_manager.cc:1398] T 0d8667d91fd64809b635362c4a8fa33f P 7ab6d395b30d4afb9821b61c9454430b: Time spent starting tablet: real 0.059s user 0.002s sys 0.000s
I20240502 14:26:51.175829 18854 tablet_copy_service.cc:342] P fec04c50dc5b49878ffa3c3873df32e4: Request end of tablet copy session 7ab6d395b30d4afb9821b61c9454430b-0d8667d91fd64809b635362c4a8fa33f received from {username='slave'} at 127.5.107.71:44135
I20240502 14:26:51.176043 18854 tablet_copy_service.cc:433] P fec04c50dc5b49878ffa3c3873df32e4: ending tablet copy session 7ab6d395b30d4afb9821b61c9454430b-0d8667d91fd64809b635362c4a8fa33f on tablet 0d8667d91fd64809b635362c4a8fa33f with peer 7ab6d395b30d4afb9821b61c9454430b
I20240502 14:26:51.181069 22073 tablet_bootstrap.cc:492] T 0ffb6e582ef04fb38dd273c13febac58 P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:51.181679 22073 tablet_bootstrap.cc:492] T 0ffb6e582ef04fb38dd273c13febac58 P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap complete.
I20240502 14:26:51.182019 22073 ts_tablet_manager.cc:1367] T 0ffb6e582ef04fb38dd273c13febac58 P ff6acba046404ef0a6f41ad5e09d852f: Time spent bootstrapping tablet: real 1.014s user 0.064s sys 0.000s
I20240502 14:26:51.183048 22073 raft_consensus.cc:348] T 0ffb6e582ef04fb38dd273c13febac58 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:26:51.183351 22073 raft_consensus.cc:729] T 0ffb6e582ef04fb38dd273c13febac58 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6acba046404ef0a6f41ad5e09d852f, State: Initialized, Role: FOLLOWER
I20240502 14:26:51.183717 22073 consensus_queue.cc:260] T 0ffb6e582ef04fb38dd273c13febac58 P ff6acba046404ef0a6f41ad5e09d852f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:26:51.184449 22280 tablet_copy_client.cc:783] T 8937c809610542d88fdde3daf002c890 P a834edffbcb2469aae598b069aab0681: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:51.184818 22280 tablet_copy_client.cc:647] T 8937c809610542d88fdde3daf002c890 P a834edffbcb2469aae598b069aab0681: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:51.185333 22073 ts_tablet_manager.cc:1398] T 0ffb6e582ef04fb38dd273c13febac58 P ff6acba046404ef0a6f41ad5e09d852f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:26:51.185905 18833 raft_consensus.cc:1261] T b315327141954b53a0f87c2d1dc14b9b P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:51.187019 22248 consensus_queue.cc:1035] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:51.188187 18721 tablet_copy_service.cc:342] P 2db304b538e14857b7b1cc272d0db94b: Request end of tablet copy session ff6acba046404ef0a6f41ad5e09d852f-0ffb6e582ef04fb38dd273c13febac58 received from {username='slave'} at 127.5.107.72:37745
I20240502 14:26:51.188402 18721 tablet_copy_service.cc:433] P 2db304b538e14857b7b1cc272d0db94b: ending tablet copy session ff6acba046404ef0a6f41ad5e09d852f-0ffb6e582ef04fb38dd273c13febac58 on tablet 0ffb6e582ef04fb38dd273c13febac58 with peer ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:26:51.190559 21791 raft_consensus.cc:2932] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:51.198247 18832 raft_consensus.cc:2932] T b315327141954b53a0f87c2d1dc14b9b P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:51.199244 19093 raft_consensus.cc:2932] T b315327141954b53a0f87c2d1dc14b9b P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:51.194376 21029 raft_consensus.cc:2932] T 8937c809610542d88fdde3daf002c890 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:51.224629 22280 tablet_copy_client.cc:523] T 8937c809610542d88fdde3daf002c890 P a834edffbcb2469aae598b069aab0681: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:51.224515 18558 catalog_manager.cc:5484] T b315327141954b53a0f87c2d1dc14b9b P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: config changed from index 4 to 5, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:51.239750 20894 raft_consensus.cc:1203] T cf2a6426ae6c438f942d39530da6e3e3 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:26:51.279604 21029 raft_consensus.cc:1203] T 0ffb6e582ef04fb38dd273c13febac58 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:26:51.283114 22280 tablet_bootstrap.cc:492] T 8937c809610542d88fdde3daf002c890 P a834edffbcb2469aae598b069aab0681: Bootstrap starting.
W20240502 14:26:51.284487 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b315327141954b53a0f87c2d1dc14b9b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:51.358870 18700 consensus_queue.cc:237] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:51.372006 18965 raft_consensus.cc:1261] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:51.372714 21791 consensus_queue.cc:1035] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:51.408218 19093 raft_consensus.cc:1261] T df5f14206a1d46e4866f34fa234f945b P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:51.411101 21791 consensus_queue.cc:1035] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:51.414049 22280 tablet_bootstrap.cc:492] T 8937c809610542d88fdde3daf002c890 P a834edffbcb2469aae598b069aab0681: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:51.414649 22280 tablet_bootstrap.cc:492] T 8937c809610542d88fdde3daf002c890 P a834edffbcb2469aae598b069aab0681: Bootstrap complete.
I20240502 14:26:51.415007 22280 ts_tablet_manager.cc:1367] T 8937c809610542d88fdde3daf002c890 P a834edffbcb2469aae598b069aab0681: Time spent bootstrapping tablet: real 0.132s user 0.064s sys 0.004s
I20240502 14:26:51.416519 20760 raft_consensus.cc:1203] T e5fb3b049ba94af998791640bd0e31c5 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:26:51.416368 22280 raft_consensus.cc:348] T 8937c809610542d88fdde3daf002c890 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:51.416687 22280 raft_consensus.cc:729] T 8937c809610542d88fdde3daf002c890 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a834edffbcb2469aae598b069aab0681, State: Initialized, Role: FOLLOWER
I20240502 14:26:51.416973 22280 consensus_queue.cc:260] T 8937c809610542d88fdde3daf002c890 P a834edffbcb2469aae598b069aab0681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:51.431715 22073 ts_tablet_manager.cc:902] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f: Initiating tablet copy from peer fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
I20240502 14:26:51.449133 21791 raft_consensus.cc:2932] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:51.458159 18965 raft_consensus.cc:2932] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:51.459432 18557 catalog_manager.cc:5484] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b reported cstate change: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:51.466699 19093 raft_consensus.cc:2932] T df5f14206a1d46e4866f34fa234f945b P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
W20240502 14:26:51.483995 18637 consensus_peers.cc:485] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df5f14206a1d46e4866f34fa234f945b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:51.487279 20894 raft_consensus.cc:1203] T 0d8667d91fd64809b635362c4a8fa33f P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:26:51.501705 22280 ts_tablet_manager.cc:1398] T 8937c809610542d88fdde3daf002c890 P a834edffbcb2469aae598b069aab0681: Time spent starting tablet: real 0.087s user 0.019s sys 0.044s
I20240502 14:26:51.502571 19117 tablet_copy_service.cc:342] P 007b2e79dc424a6f80141180f072f2d8: Request end of tablet copy session a834edffbcb2469aae598b069aab0681-8937c809610542d88fdde3daf002c890 received from {username='slave'} at 127.5.107.69:37525
I20240502 14:26:51.502718 19117 tablet_copy_service.cc:433] P 007b2e79dc424a6f80141180f072f2d8: ending tablet copy session a834edffbcb2469aae598b069aab0681-8937c809610542d88fdde3daf002c890 on tablet 8937c809610542d88fdde3daf002c890 with peer a834edffbcb2469aae598b069aab0681
I20240502 14:26:51.522089 22073 tablet_copy_client.cc:308] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:33009
I20240502 14:26:51.524158 18854 tablet_copy_service.cc:140] P fec04c50dc5b49878ffa3c3873df32e4: Received BeginTabletCopySession request for tablet 522014370f0a4cc3bd31aa00d6a05c5e from peer ff6acba046404ef0a6f41ad5e09d852f ({username='slave'} at 127.5.107.72:58697)
I20240502 14:26:51.524398 18854 tablet_copy_service.cc:161] P fec04c50dc5b49878ffa3c3873df32e4: Beginning new tablet copy session on tablet 522014370f0a4cc3bd31aa00d6a05c5e from peer ff6acba046404ef0a6f41ad5e09d852f at {username='slave'} at 127.5.107.72:58697: session id = ff6acba046404ef0a6f41ad5e09d852f-522014370f0a4cc3bd31aa00d6a05c5e
I20240502 14:26:51.527447 18854 tablet_copy_source_session.cc:205] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:51.529052 22073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 522014370f0a4cc3bd31aa00d6a05c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:51.536092 20626 raft_consensus.cc:1203] T 8937c809610542d88fdde3daf002c890 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:26:51.546694 22073 tablet_copy_client.cc:783] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:51.546981 22073 tablet_copy_client.cc:647] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:51.549067 22073 tablet_copy_client.cc:523] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:51.554718 22073 tablet_bootstrap.cc:492] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap starting.
I20240502 14:26:51.572275 18832 consensus_queue.cc:237] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:51.583034 18965 raft_consensus.cc:1261] T 8b8d81c275144f898dcc69a6d98d11e0 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:26:51.583582 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8b8d81c275144f898dcc69a6d98d11e0. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:51.584157 21718 consensus_queue.cc:1035] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:51.585662 18700 raft_consensus.cc:1261] T 8b8d81c275144f898dcc69a6d98d11e0 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:51.587689 21718 consensus_queue.cc:1035] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:51.592905 22395 ts_tablet_manager.cc:902] T b315327141954b53a0f87c2d1dc14b9b P a14c220ad2cc4bf9bd7dad65f4803eb0: Initiating tablet copy from peer 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:26:51.615149 21718 raft_consensus.cc:2932] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:51.620682 18700 raft_consensus.cc:2932] T 8b8d81c275144f898dcc69a6d98d11e0 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:51.636626 22395 tablet_copy_client.cc:308] T b315327141954b53a0f87c2d1dc14b9b P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.65:36749
I20240502 14:26:51.640136 18965 raft_consensus.cc:2932] T 8b8d81c275144f898dcc69a6d98d11e0 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:51.649044 18558 catalog_manager.cc:5484] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index -1 to 4, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:51.679512 18721 tablet_copy_service.cc:140] P 2db304b538e14857b7b1cc272d0db94b: Received BeginTabletCopySession request for tablet b315327141954b53a0f87c2d1dc14b9b from peer a14c220ad2cc4bf9bd7dad65f4803eb0 ({username='slave'} at 127.5.107.70:48397)
I20240502 14:26:51.679781 18721 tablet_copy_service.cc:161] P 2db304b538e14857b7b1cc272d0db94b: Beginning new tablet copy session on tablet b315327141954b53a0f87c2d1dc14b9b from peer a14c220ad2cc4bf9bd7dad65f4803eb0 at {username='slave'} at 127.5.107.70:48397: session id = a14c220ad2cc4bf9bd7dad65f4803eb0-b315327141954b53a0f87c2d1dc14b9b
I20240502 14:26:51.683624 18721 tablet_copy_source_session.cc:205] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:51.685002 22395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b315327141954b53a0f87c2d1dc14b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:51.691480 22395 tablet_copy_client.cc:783] T b315327141954b53a0f87c2d1dc14b9b P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:51.691780 22395 tablet_copy_client.cc:647] T b315327141954b53a0f87c2d1dc14b9b P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:51.696978 22395 tablet_copy_client.cc:523] T b315327141954b53a0f87c2d1dc14b9b P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:51.710253 22395 tablet_bootstrap.cc:492] T b315327141954b53a0f87c2d1dc14b9b P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap starting.
I20240502 14:26:51.730742 22073 tablet_bootstrap.cc:492] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:51.736259 18832 consensus_queue.cc:237] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:51.737129 22073 tablet_bootstrap.cc:492] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap complete.
I20240502 14:26:51.737481 22073 ts_tablet_manager.cc:1367] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f: Time spent bootstrapping tablet: real 0.183s user 0.068s sys 0.000s
I20240502 14:26:51.738351 22073 raft_consensus.cc:348] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:26:51.738965 22073 raft_consensus.cc:729] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6acba046404ef0a6f41ad5e09d852f, State: Initialized, Role: FOLLOWER
I20240502 14:26:51.739423 22073 consensus_queue.cc:260] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:26:51.753506 22073 ts_tablet_manager.cc:1398] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240502 14:26:51.755312 18854 tablet_copy_service.cc:342] P fec04c50dc5b49878ffa3c3873df32e4: Request end of tablet copy session ff6acba046404ef0a6f41ad5e09d852f-522014370f0a4cc3bd31aa00d6a05c5e received from {username='slave'} at 127.5.107.72:58697
I20240502 14:26:51.755457 18854 tablet_copy_service.cc:433] P fec04c50dc5b49878ffa3c3873df32e4: ending tablet copy session ff6acba046404ef0a6f41ad5e09d852f-522014370f0a4cc3bd31aa00d6a05c5e on tablet 522014370f0a4cc3bd31aa00d6a05c5e with peer ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:26:51.757609 22073 ts_tablet_manager.cc:902] T b315327141954b53a0f87c2d1dc14b9b P ff6acba046404ef0a6f41ad5e09d852f: Initiating tablet copy from peer 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:26:51.764237 19093 raft_consensus.cc:1261] T 8a2bed29fcd64f7bbb04286404ef09de P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:26:51.764935 18769 consensus_peers.cc:485] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a2bed29fcd64f7bbb04286404ef09de. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:51.765144 22396 consensus_queue.cc:1035] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:51.769160 18965 raft_consensus.cc:1261] T 8a2bed29fcd64f7bbb04286404ef09de P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:51.769826 22396 consensus_queue.cc:1035] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:51.791795 22396 raft_consensus.cc:2932] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:51.793574 18965 raft_consensus.cc:2932] T 8a2bed29fcd64f7bbb04286404ef09de P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:51.794786 22073 tablet_copy_client.cc:308] T b315327141954b53a0f87c2d1dc14b9b P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.65:36749
I20240502 14:26:51.795707 18721 tablet_copy_service.cc:140] P 2db304b538e14857b7b1cc272d0db94b: Received BeginTabletCopySession request for tablet b315327141954b53a0f87c2d1dc14b9b from peer ff6acba046404ef0a6f41ad5e09d852f ({username='slave'} at 127.5.107.72:37745)
I20240502 14:26:51.795890 18721 tablet_copy_service.cc:161] P 2db304b538e14857b7b1cc272d0db94b: Beginning new tablet copy session on tablet b315327141954b53a0f87c2d1dc14b9b from peer ff6acba046404ef0a6f41ad5e09d852f at {username='slave'} at 127.5.107.72:37745: session id = ff6acba046404ef0a6f41ad5e09d852f-b315327141954b53a0f87c2d1dc14b9b
I20240502 14:26:51.798743 18721 tablet_copy_source_session.cc:205] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:51.801577 22073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b315327141954b53a0f87c2d1dc14b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:51.823279 18558 catalog_manager.cc:5484] T 8a2bed29fcd64f7bbb04286404ef09de P c902805060df4df9b5599d0e94b2dc06 reported cstate change: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:51.830221 19076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:26:51.835759 22073 tablet_copy_client.cc:783] T b315327141954b53a0f87c2d1dc14b9b P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:51.836027 22073 tablet_copy_client.cc:647] T b315327141954b53a0f87c2d1dc14b9b P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:51.837934 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:26:51.849081 22073 tablet_copy_client.cc:523] T b315327141954b53a0f87c2d1dc14b9b P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:51.853440 22073 tablet_bootstrap.cc:492] T b315327141954b53a0f87c2d1dc14b9b P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap starting.
I20240502 14:26:51.887873 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:51.891827 19093 raft_consensus.cc:2932] T 8a2bed29fcd64f7bbb04286404ef09de P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:51.926357 22395 tablet_bootstrap.cc:492] T b315327141954b53a0f87c2d1dc14b9b P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:51.926829 22395 tablet_bootstrap.cc:492] T b315327141954b53a0f87c2d1dc14b9b P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap complete.
I20240502 14:26:51.927115 22395 ts_tablet_manager.cc:1367] T b315327141954b53a0f87c2d1dc14b9b P a14c220ad2cc4bf9bd7dad65f4803eb0: Time spent bootstrapping tablet: real 0.217s user 0.053s sys 0.017s
I20240502 14:26:51.928045 22395 raft_consensus.cc:348] T b315327141954b53a0f87c2d1dc14b9b P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:26:51.928376 22395 raft_consensus.cc:729] T b315327141954b53a0f87c2d1dc14b9b P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a14c220ad2cc4bf9bd7dad65f4803eb0, State: Initialized, Role: FOLLOWER
I20240502 14:26:51.934510 22395 consensus_queue.cc:260] T b315327141954b53a0f87c2d1dc14b9b P a14c220ad2cc4bf9bd7dad65f4803eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:26:51.946615 18832 consensus_queue.cc:237] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:26:51.947758 22395 ts_tablet_manager.cc:1398] T b315327141954b53a0f87c2d1dc14b9b P a14c220ad2cc4bf9bd7dad65f4803eb0: Time spent starting tablet: real 0.021s user 0.005s sys 0.002s
I20240502 14:26:51.948979 18721 tablet_copy_service.cc:342] P 2db304b538e14857b7b1cc272d0db94b: Request end of tablet copy session a14c220ad2cc4bf9bd7dad65f4803eb0-b315327141954b53a0f87c2d1dc14b9b received from {username='slave'} at 127.5.107.70:48397
I20240502 14:26:51.949218 18721 tablet_copy_service.cc:433] P 2db304b538e14857b7b1cc272d0db94b: ending tablet copy session a14c220ad2cc4bf9bd7dad65f4803eb0-b315327141954b53a0f87c2d1dc14b9b on tablet b315327141954b53a0f87c2d1dc14b9b with peer a14c220ad2cc4bf9bd7dad65f4803eb0
I20240502 14:26:51.951442 22280 ts_tablet_manager.cc:902] T df5f14206a1d46e4866f34fa234f945b P a834edffbcb2469aae598b069aab0681: Initiating tablet copy from peer 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:26:52.014070 18965 consensus_queue.cc:237] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:26:52.023488 21029 raft_consensus.cc:1203] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:26:52.030198 22073 tablet_bootstrap.cc:492] T b315327141954b53a0f87c2d1dc14b9b P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:52.030716 22073 tablet_bootstrap.cc:492] T b315327141954b53a0f87c2d1dc14b9b P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap complete.
I20240502 14:26:52.031031 22073 ts_tablet_manager.cc:1367] T b315327141954b53a0f87c2d1dc14b9b P ff6acba046404ef0a6f41ad5e09d852f: Time spent bootstrapping tablet: real 0.178s user 0.061s sys 0.000s
I20240502 14:26:52.034932 22073 raft_consensus.cc:348] T b315327141954b53a0f87c2d1dc14b9b P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:26:52.035238 22073 raft_consensus.cc:729] T b315327141954b53a0f87c2d1dc14b9b P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6acba046404ef0a6f41ad5e09d852f, State: Initialized, Role: FOLLOWER
I20240502 14:26:52.038545 22073 consensus_queue.cc:260] T b315327141954b53a0f87c2d1dc14b9b P ff6acba046404ef0a6f41ad5e09d852f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:26:52.039924 22073 ts_tablet_manager.cc:1398] T b315327141954b53a0f87c2d1dc14b9b P ff6acba046404ef0a6f41ad5e09d852f: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20240502 14:26:52.041841 18721 tablet_copy_service.cc:342] P 2db304b538e14857b7b1cc272d0db94b: Request end of tablet copy session ff6acba046404ef0a6f41ad5e09d852f-b315327141954b53a0f87c2d1dc14b9b received from {username='slave'} at 127.5.107.72:37745
I20240502 14:26:52.041987 18721 tablet_copy_service.cc:433] P 2db304b538e14857b7b1cc272d0db94b: ending tablet copy session ff6acba046404ef0a6f41ad5e09d852f-b315327141954b53a0f87c2d1dc14b9b on tablet b315327141954b53a0f87c2d1dc14b9b with peer ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:26:52.052078 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:52.061664 22470 ts_tablet_manager.cc:902] T 8b8d81c275144f898dcc69a6d98d11e0 P 7ab6d395b30d4afb9821b61c9454430b: Initiating tablet copy from peer fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
I20240502 14:26:52.073680 20760 raft_consensus.cc:1203] T b315327141954b53a0f87c2d1dc14b9b P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240502 14:26:52.080607 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d8667d91fd64809b635362c4a8fa33f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:52.082334 20894 raft_consensus.cc:1261] T 0d8667d91fd64809b635362c4a8fa33f P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:52.083258 21718 consensus_queue.cc:1035] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:52.089223 19093 raft_consensus.cc:1261] T 0d8667d91fd64809b635362c4a8fa33f P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:52.090054 21718 consensus_queue.cc:1035] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:52.096660 18701 raft_consensus.cc:1261] T 0d8667d91fd64809b635362c4a8fa33f P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:52.097357 21718 consensus_queue.cc:1035] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:52.116999 19097 raft_consensus.cc:1261] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:52.118350 21881 consensus_queue.cc:1035] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:52.130668 19095 consensus_queue.cc:237] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:26:52.146982 18832 raft_consensus.cc:1261] T 21af917743444f2db932636a78f33512 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:52.147929 21881 consensus_queue.cc:1035] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:52.197647 22280 tablet_copy_client.cc:308] T df5f14206a1d46e4866f34fa234f945b P a834edffbcb2469aae598b069aab0681: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.65:36749
I20240502 14:26:52.222915 18700 raft_consensus.cc:2932] T 0d8667d91fd64809b635362c4a8fa33f P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:52.225519 22396 raft_consensus.cc:2932] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:52.256115 22493 raft_consensus.cc:2932] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:52.257220 18832 raft_consensus.cc:2932] T 21af917743444f2db932636a78f33512 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:52.292629 19097 raft_consensus.cc:2932] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:52.298375 18557 catalog_manager.cc:5484] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 reported cstate change: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:52.302441 18558 catalog_manager.cc:5484] T 0d8667d91fd64809b635362c4a8fa33f P 2db304b538e14857b7b1cc272d0db94b reported cstate change: config changed from index 4 to 5, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:52.286193 20894 raft_consensus.cc:2932] T 0d8667d91fd64809b635362c4a8fa33f P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:52.319073 19093 raft_consensus.cc:2932] T 0d8667d91fd64809b635362c4a8fa33f P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:52.330399 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:52.340149 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:52.351428 21029 raft_consensus.cc:1203] T b315327141954b53a0f87c2d1dc14b9b P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240502 14:26:52.392578 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21af917743444f2db932636a78f33512. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:52.394538 22470 tablet_copy_client.cc:308] T 8b8d81c275144f898dcc69a6d98d11e0 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:33009
I20240502 14:26:52.395380 18854 tablet_copy_service.cc:140] P fec04c50dc5b49878ffa3c3873df32e4: Received BeginTabletCopySession request for tablet 8b8d81c275144f898dcc69a6d98d11e0 from peer 7ab6d395b30d4afb9821b61c9454430b ({username='slave'} at 127.5.107.71:44135)
I20240502 14:26:52.395571 18854 tablet_copy_service.cc:161] P fec04c50dc5b49878ffa3c3873df32e4: Beginning new tablet copy session on tablet 8b8d81c275144f898dcc69a6d98d11e0 from peer 7ab6d395b30d4afb9821b61c9454430b at {username='slave'} at 127.5.107.71:44135: session id = 7ab6d395b30d4afb9821b61c9454430b-8b8d81c275144f898dcc69a6d98d11e0
I20240502 14:26:52.399250 18854 tablet_copy_source_session.cc:205] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:52.400900 22470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b8d81c275144f898dcc69a6d98d11e0. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 14:26:52.410701 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bad03a4ec8494938839d7d6eef8c684e. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:52.411226 18965 raft_consensus.cc:1261] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:52.411382 22470 tablet_copy_client.cc:783] T 8b8d81c275144f898dcc69a6d98d11e0 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:52.411640 22470 tablet_copy_client.cc:647] T 8b8d81c275144f898dcc69a6d98d11e0 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:52.411844 22507 consensus_queue.cc:1035] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:52.412678 18701 raft_consensus.cc:1261] T bad03a4ec8494938839d7d6eef8c684e P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:52.413256 22232 consensus_queue.cc:1035] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:52.419652 22505 ts_tablet_manager.cc:902] T 8a2bed29fcd64f7bbb04286404ef09de P a834edffbcb2469aae598b069aab0681: Initiating tablet copy from peer fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
I20240502 14:26:52.425971 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:26:52.426852 22470 tablet_copy_client.cc:523] T 8b8d81c275144f898dcc69a6d98d11e0 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:52.431869 21007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:52.434952 22470 tablet_bootstrap.cc:492] T 8b8d81c275144f898dcc69a6d98d11e0 P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap starting.
I20240502 14:26:52.436079 18721 tablet_copy_service.cc:140] P 2db304b538e14857b7b1cc272d0db94b: Received BeginTabletCopySession request for tablet df5f14206a1d46e4866f34fa234f945b from peer a834edffbcb2469aae598b069aab0681 ({username='slave'} at 127.5.107.69:53895)
I20240502 14:26:52.436229 18721 tablet_copy_service.cc:161] P 2db304b538e14857b7b1cc272d0db94b: Beginning new tablet copy session on tablet df5f14206a1d46e4866f34fa234f945b from peer a834edffbcb2469aae598b069aab0681 at {username='slave'} at 127.5.107.69:53895: session id = a834edffbcb2469aae598b069aab0681-df5f14206a1d46e4866f34fa234f945b
I20240502 14:26:52.439100 18721 tablet_copy_source_session.cc:205] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:52.440512 22280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df5f14206a1d46e4866f34fa234f945b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:52.446803 22280 tablet_copy_client.cc:783] T df5f14206a1d46e4866f34fa234f945b P a834edffbcb2469aae598b069aab0681: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:52.447008 22280 tablet_copy_client.cc:647] T df5f14206a1d46e4866f34fa234f945b P a834edffbcb2469aae598b069aab0681: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:52.454315 22340 raft_consensus.cc:2932] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:52.458247 18964 raft_consensus.cc:2932] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:52.466392 18558 catalog_manager.cc:5484] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06 reported cstate change: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:52.471181 22505 tablet_copy_client.cc:308] T 8a2bed29fcd64f7bbb04286404ef09de P a834edffbcb2469aae598b069aab0681: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:33009
I20240502 14:26:52.472419 18854 tablet_copy_service.cc:140] P fec04c50dc5b49878ffa3c3873df32e4: Received BeginTabletCopySession request for tablet 8a2bed29fcd64f7bbb04286404ef09de from peer a834edffbcb2469aae598b069aab0681 ({username='slave'} at 127.5.107.69:37387)
I20240502 14:26:52.472622 18854 tablet_copy_service.cc:161] P fec04c50dc5b49878ffa3c3873df32e4: Beginning new tablet copy session on tablet 8a2bed29fcd64f7bbb04286404ef09de from peer a834edffbcb2469aae598b069aab0681 at {username='slave'} at 127.5.107.69:37387: session id = a834edffbcb2469aae598b069aab0681-8a2bed29fcd64f7bbb04286404ef09de
I20240502 14:26:52.474002 22073 ts_tablet_manager.cc:902] T 0d8667d91fd64809b635362c4a8fa33f P ff6acba046404ef0a6f41ad5e09d852f: Initiating tablet copy from peer fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
I20240502 14:26:52.474242 22280 tablet_copy_client.cc:523] T df5f14206a1d46e4866f34fa234f945b P a834edffbcb2469aae598b069aab0681: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:52.476030 18854 tablet_copy_source_session.cc:205] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:52.477350 22505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a2bed29fcd64f7bbb04286404ef09de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:52.488983 22505 tablet_copy_client.cc:783] T 8a2bed29fcd64f7bbb04286404ef09de P a834edffbcb2469aae598b069aab0681: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:52.489383 22505 tablet_copy_client.cc:647] T 8a2bed29fcd64f7bbb04286404ef09de P a834edffbcb2469aae598b069aab0681: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:52.491669 22505 tablet_copy_client.cc:523] T 8a2bed29fcd64f7bbb04286404ef09de P a834edffbcb2469aae598b069aab0681: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:52.489578 18700 raft_consensus.cc:2932] T bad03a4ec8494938839d7d6eef8c684e P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:52.499199 22280 tablet_bootstrap.cc:492] T df5f14206a1d46e4866f34fa234f945b P a834edffbcb2469aae598b069aab0681: Bootstrap starting.
I20240502 14:26:52.502549 22505 tablet_bootstrap.cc:492] T 8a2bed29fcd64f7bbb04286404ef09de P a834edffbcb2469aae598b069aab0681: Bootstrap starting.
I20240502 14:26:52.534047 22073 tablet_copy_client.cc:308] T 0d8667d91fd64809b635362c4a8fa33f P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:33009
I20240502 14:26:52.534724 18854 tablet_copy_service.cc:140] P fec04c50dc5b49878ffa3c3873df32e4: Received BeginTabletCopySession request for tablet 0d8667d91fd64809b635362c4a8fa33f from peer ff6acba046404ef0a6f41ad5e09d852f ({username='slave'} at 127.5.107.72:58697)
I20240502 14:26:52.534888 18854 tablet_copy_service.cc:161] P fec04c50dc5b49878ffa3c3873df32e4: Beginning new tablet copy session on tablet 0d8667d91fd64809b635362c4a8fa33f from peer ff6acba046404ef0a6f41ad5e09d852f at {username='slave'} at 127.5.107.72:58697: session id = ff6acba046404ef0a6f41ad5e09d852f-0d8667d91fd64809b635362c4a8fa33f
I20240502 14:26:52.538259 18854 tablet_copy_source_session.cc:205] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:52.539661 22073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d8667d91fd64809b635362c4a8fa33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:52.584587 22544 ts_tablet_manager.cc:902] T 21af917743444f2db932636a78f33512 P a14c220ad2cc4bf9bd7dad65f4803eb0: Initiating tablet copy from peer c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:26:52.598745 22073 tablet_copy_client.cc:783] T 0d8667d91fd64809b635362c4a8fa33f P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:52.599105 22073 tablet_copy_client.cc:647] T 0d8667d91fd64809b635362c4a8fa33f P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:52.602699 22073 tablet_copy_client.cc:523] T 0d8667d91fd64809b635362c4a8fa33f P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:52.606137 22073 tablet_bootstrap.cc:492] T 0d8667d91fd64809b635362c4a8fa33f P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap starting.
I20240502 14:26:52.644951 18832 consensus_queue.cc:237] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:52.648181 19097 raft_consensus.cc:1261] T 414e03d597934da3a6a74d786aeab134 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:26:52.648705 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 414e03d597934da3a6a74d786aeab134. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:52.648798 18965 raft_consensus.cc:1261] T 414e03d597934da3a6a74d786aeab134 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:52.649613 22501 consensus_queue.cc:1035] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:52.649891 22505 tablet_bootstrap.cc:492] T 8a2bed29fcd64f7bbb04286404ef09de P a834edffbcb2469aae598b069aab0681: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:52.650231 22501 consensus_queue.cc:1035] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:52.650398 22505 tablet_bootstrap.cc:492] T 8a2bed29fcd64f7bbb04286404ef09de P a834edffbcb2469aae598b069aab0681: Bootstrap complete.
I20240502 14:26:52.650691 22505 ts_tablet_manager.cc:1367] T 8a2bed29fcd64f7bbb04286404ef09de P a834edffbcb2469aae598b069aab0681: Time spent bootstrapping tablet: real 0.148s user 0.048s sys 0.005s
I20240502 14:26:52.651477 22505 raft_consensus.cc:348] T 8a2bed29fcd64f7bbb04286404ef09de P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:52.651782 22505 raft_consensus.cc:729] T 8a2bed29fcd64f7bbb04286404ef09de P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a834edffbcb2469aae598b069aab0681, State: Initialized, Role: FOLLOWER
I20240502 14:26:52.655315 22505 consensus_queue.cc:260] T 8a2bed29fcd64f7bbb04286404ef09de P a834edffbcb2469aae598b069aab0681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:52.657773 22500 raft_consensus.cc:2932] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:52.663314 22505 ts_tablet_manager.cc:1398] T 8a2bed29fcd64f7bbb04286404ef09de P a834edffbcb2469aae598b069aab0681: Time spent starting tablet: real 0.013s user 0.002s sys 0.001s
I20240502 14:26:52.663122 19097 raft_consensus.cc:2932] T 414e03d597934da3a6a74d786aeab134 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:52.664800 18854 tablet_copy_service.cc:342] P fec04c50dc5b49878ffa3c3873df32e4: Request end of tablet copy session a834edffbcb2469aae598b069aab0681-8a2bed29fcd64f7bbb04286404ef09de received from {username='slave'} at 127.5.107.69:37387
I20240502 14:26:52.664916 18854 tablet_copy_service.cc:433] P fec04c50dc5b49878ffa3c3873df32e4: ending tablet copy session a834edffbcb2469aae598b069aab0681-8a2bed29fcd64f7bbb04286404ef09de on tablet 8a2bed29fcd64f7bbb04286404ef09de with peer a834edffbcb2469aae598b069aab0681
I20240502 14:26:52.666345 18965 raft_consensus.cc:2932] T 414e03d597934da3a6a74d786aeab134 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:52.679426 18558 catalog_manager.cc:5484] T 414e03d597934da3a6a74d786aeab134 P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: config changed from index -1 to 4, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:52.694710 22544 tablet_copy_client.cc:308] T 21af917743444f2db932636a78f33512 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.67:45409
I20240502 14:26:52.699035 22280 tablet_bootstrap.cc:492] T df5f14206a1d46e4866f34fa234f945b P a834edffbcb2469aae598b069aab0681: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:52.699563 22280 tablet_bootstrap.cc:492] T df5f14206a1d46e4866f34fa234f945b P a834edffbcb2469aae598b069aab0681: Bootstrap complete.
I20240502 14:26:52.699862 22280 ts_tablet_manager.cc:1367] T df5f14206a1d46e4866f34fa234f945b P a834edffbcb2469aae598b069aab0681: Time spent bootstrapping tablet: real 0.201s user 0.053s sys 0.002s
I20240502 14:26:52.700928 22280 raft_consensus.cc:348] T df5f14206a1d46e4866f34fa234f945b P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:52.701228 22280 raft_consensus.cc:729] T df5f14206a1d46e4866f34fa234f945b P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a834edffbcb2469aae598b069aab0681, State: Initialized, Role: FOLLOWER
I20240502 14:26:52.701175 18832 consensus_queue.cc:237] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:52.701507 22280 consensus_queue.cc:260] T df5f14206a1d46e4866f34fa234f945b P a834edffbcb2469aae598b069aab0681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:52.703810 22470 tablet_bootstrap.cc:492] T 8b8d81c275144f898dcc69a6d98d11e0 P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:52.704290 22470 tablet_bootstrap.cc:492] T 8b8d81c275144f898dcc69a6d98d11e0 P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap complete.
I20240502 14:26:52.704617 22470 ts_tablet_manager.cc:1367] T 8b8d81c275144f898dcc69a6d98d11e0 P 7ab6d395b30d4afb9821b61c9454430b: Time spent bootstrapping tablet: real 0.270s user 0.050s sys 0.005s
I20240502 14:26:52.705539 22470 raft_consensus.cc:348] T 8b8d81c275144f898dcc69a6d98d11e0 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:52.705780 22470 raft_consensus.cc:729] T 8b8d81c275144f898dcc69a6d98d11e0 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab6d395b30d4afb9821b61c9454430b, State: Initialized, Role: FOLLOWER
I20240502 14:26:52.712293 21029 raft_consensus.cc:1261] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:52.712388 22280 ts_tablet_manager.cc:1398] T df5f14206a1d46e4866f34fa234f945b P a834edffbcb2469aae598b069aab0681: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 14:26:52.713074 18721 tablet_copy_service.cc:342] P 2db304b538e14857b7b1cc272d0db94b: Request end of tablet copy session a834edffbcb2469aae598b069aab0681-df5f14206a1d46e4866f34fa234f945b received from {username='slave'} at 127.5.107.69:53895
I20240502 14:26:52.713045 22490 consensus_queue.cc:1035] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:52.713243 18721 tablet_copy_service.cc:433] P 2db304b538e14857b7b1cc272d0db94b: ending tablet copy session a834edffbcb2469aae598b069aab0681-df5f14206a1d46e4866f34fa234f945b on tablet df5f14206a1d46e4866f34fa234f945b with peer a834edffbcb2469aae598b069aab0681
I20240502 14:26:52.712752 22470 consensus_queue.cc:260] T 8b8d81c275144f898dcc69a6d98d11e0 P 7ab6d395b30d4afb9821b61c9454430b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:52.715775 18985 tablet_copy_service.cc:140] P c902805060df4df9b5599d0e94b2dc06: Received BeginTabletCopySession request for tablet 21af917743444f2db932636a78f33512 from peer a14c220ad2cc4bf9bd7dad65f4803eb0 ({username='slave'} at 127.5.107.70:56629)
I20240502 14:26:52.715932 18985 tablet_copy_service.cc:161] P c902805060df4df9b5599d0e94b2dc06: Beginning new tablet copy session on tablet 21af917743444f2db932636a78f33512 from peer a14c220ad2cc4bf9bd7dad65f4803eb0 at {username='slave'} at 127.5.107.70:56629: session id = a14c220ad2cc4bf9bd7dad65f4803eb0-21af917743444f2db932636a78f33512
I20240502 14:26:52.716859 22470 ts_tablet_manager.cc:1398] T 8b8d81c275144f898dcc69a6d98d11e0 P 7ab6d395b30d4afb9821b61c9454430b: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 14:26:52.718636 18985 tablet_copy_source_session.cc:205] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:52.719478 18854 tablet_copy_service.cc:342] P fec04c50dc5b49878ffa3c3873df32e4: Request end of tablet copy session 7ab6d395b30d4afb9821b61c9454430b-8b8d81c275144f898dcc69a6d98d11e0 received from {username='slave'} at 127.5.107.71:44135
I20240502 14:26:52.719627 18854 tablet_copy_service.cc:433] P fec04c50dc5b49878ffa3c3873df32e4: ending tablet copy session 7ab6d395b30d4afb9821b61c9454430b-8b8d81c275144f898dcc69a6d98d11e0 on tablet 8b8d81c275144f898dcc69a6d98d11e0 with peer 7ab6d395b30d4afb9821b61c9454430b
W20240502 14:26:52.721763 18769 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 522014370f0a4cc3bd31aa00d6a05c5e. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:52.722398 19097 raft_consensus.cc:1261] T 522014370f0a4cc3bd31aa00d6a05c5e P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:52.723037 22544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21af917743444f2db932636a78f33512. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:52.723079 22500 consensus_queue.cc:1035] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:52.725772 18700 raft_consensus.cc:1261] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:52.729786 22490 consensus_queue.cc:1035] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:52.730089 22544 tablet_copy_client.cc:783] T 21af917743444f2db932636a78f33512 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:52.730361 22544 tablet_copy_client.cc:647] T 21af917743444f2db932636a78f33512 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:52.731927 22544 tablet_copy_client.cc:523] T 21af917743444f2db932636a78f33512 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:52.733706 22500 raft_consensus.cc:2932] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:52.735581 22544 tablet_bootstrap.cc:492] T 21af917743444f2db932636a78f33512 P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap starting.
I20240502 14:26:52.737838 18700 raft_consensus.cc:2932] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:52.744385 21029 raft_consensus.cc:2932] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:52.758960 18557 catalog_manager.cc:5484] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f reported cstate change: config changed from index 4 to 5, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:52.799419 19095 raft_consensus.cc:2932] T 522014370f0a4cc3bd31aa00d6a05c5e P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:52.873049 22073 tablet_bootstrap.cc:492] T 0d8667d91fd64809b635362c4a8fa33f P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:52.873503 22073 tablet_bootstrap.cc:492] T 0d8667d91fd64809b635362c4a8fa33f P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap complete.
I20240502 14:26:52.873893 22073 ts_tablet_manager.cc:1367] T 0d8667d91fd64809b635362c4a8fa33f P ff6acba046404ef0a6f41ad5e09d852f: Time spent bootstrapping tablet: real 0.268s user 0.075s sys 0.028s
I20240502 14:26:52.874758 22073 raft_consensus.cc:348] T 0d8667d91fd64809b635362c4a8fa33f P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:26:52.875061 22073 raft_consensus.cc:729] T 0d8667d91fd64809b635362c4a8fa33f P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6acba046404ef0a6f41ad5e09d852f, State: Initialized, Role: FOLLOWER
I20240502 14:26:52.875310 22073 consensus_queue.cc:260] T 0d8667d91fd64809b635362c4a8fa33f P ff6acba046404ef0a6f41ad5e09d852f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:26:52.875954 22073 ts_tablet_manager.cc:1398] T 0d8667d91fd64809b635362c4a8fa33f P ff6acba046404ef0a6f41ad5e09d852f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:26:52.876999 18854 tablet_copy_service.cc:342] P fec04c50dc5b49878ffa3c3873df32e4: Request end of tablet copy session ff6acba046404ef0a6f41ad5e09d852f-0d8667d91fd64809b635362c4a8fa33f received from {username='slave'} at 127.5.107.72:58697
I20240502 14:26:52.877115 18854 tablet_copy_service.cc:433] P fec04c50dc5b49878ffa3c3873df32e4: ending tablet copy session ff6acba046404ef0a6f41ad5e09d852f-0d8667d91fd64809b635362c4a8fa33f on tablet 0d8667d91fd64809b635362c4a8fa33f with peer ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:26:52.891928 22604 ts_tablet_manager.cc:902] T bad03a4ec8494938839d7d6eef8c684e P a14c220ad2cc4bf9bd7dad65f4803eb0: Initiating tablet copy from peer 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:26:52.911604 22604 tablet_copy_client.cc:308] T bad03a4ec8494938839d7d6eef8c684e P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.68:41463
I20240502 14:26:52.912278 22544 tablet_bootstrap.cc:492] T 21af917743444f2db932636a78f33512 P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:52.912384 19117 tablet_copy_service.cc:140] P 007b2e79dc424a6f80141180f072f2d8: Received BeginTabletCopySession request for tablet bad03a4ec8494938839d7d6eef8c684e from peer a14c220ad2cc4bf9bd7dad65f4803eb0 ({username='slave'} at 127.5.107.70:47193)
I20240502 14:26:52.912532 19117 tablet_copy_service.cc:161] P 007b2e79dc424a6f80141180f072f2d8: Beginning new tablet copy session on tablet bad03a4ec8494938839d7d6eef8c684e from peer a14c220ad2cc4bf9bd7dad65f4803eb0 at {username='slave'} at 127.5.107.70:47193: session id = a14c220ad2cc4bf9bd7dad65f4803eb0-bad03a4ec8494938839d7d6eef8c684e
I20240502 14:26:52.912771 22544 tablet_bootstrap.cc:492] T 21af917743444f2db932636a78f33512 P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap complete.
I20240502 14:26:52.913087 22544 ts_tablet_manager.cc:1367] T 21af917743444f2db932636a78f33512 P a14c220ad2cc4bf9bd7dad65f4803eb0: Time spent bootstrapping tablet: real 0.178s user 0.089s sys 0.060s
I20240502 14:26:52.914140 22544 raft_consensus.cc:348] T 21af917743444f2db932636a78f33512 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:26:52.914404 22544 raft_consensus.cc:729] T 21af917743444f2db932636a78f33512 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a14c220ad2cc4bf9bd7dad65f4803eb0, State: Initialized, Role: FOLLOWER
I20240502 14:26:52.914685 22544 consensus_queue.cc:260] T 21af917743444f2db932636a78f33512 P a14c220ad2cc4bf9bd7dad65f4803eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:26:52.915261 19117 tablet_copy_source_session.cc:205] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:52.916999 22604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bad03a4ec8494938839d7d6eef8c684e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:52.926393 22604 tablet_copy_client.cc:783] T bad03a4ec8494938839d7d6eef8c684e P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:52.926803 22604 tablet_copy_client.cc:647] T bad03a4ec8494938839d7d6eef8c684e P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:52.929069 22604 tablet_copy_client.cc:523] T bad03a4ec8494938839d7d6eef8c684e P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:52.934762 22604 tablet_bootstrap.cc:492] T bad03a4ec8494938839d7d6eef8c684e P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap starting.
I20240502 14:26:52.939446 22544 ts_tablet_manager.cc:1398] T 21af917743444f2db932636a78f33512 P a14c220ad2cc4bf9bd7dad65f4803eb0: Time spent starting tablet: real 0.026s user 0.005s sys 0.006s
I20240502 14:26:52.940390 18985 tablet_copy_service.cc:342] P c902805060df4df9b5599d0e94b2dc06: Request end of tablet copy session a14c220ad2cc4bf9bd7dad65f4803eb0-21af917743444f2db932636a78f33512 received from {username='slave'} at 127.5.107.70:56629
I20240502 14:26:52.940523 18985 tablet_copy_service.cc:433] P c902805060df4df9b5599d0e94b2dc06: ending tablet copy session a14c220ad2cc4bf9bd7dad65f4803eb0-21af917743444f2db932636a78f33512 on tablet 21af917743444f2db932636a78f33512 with peer a14c220ad2cc4bf9bd7dad65f4803eb0
I20240502 14:26:52.947892 20626 raft_consensus.cc:1203] T df5f14206a1d46e4866f34fa234f945b P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:26:52.966568 20626 raft_consensus.cc:1203] T 8a2bed29fcd64f7bbb04286404ef09de P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:26:52.988173 18700 consensus_queue.cc:237] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:52.997802 18701 consensus_queue.cc:237] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:53.003078 20760 raft_consensus.cc:1203] T 21af917743444f2db932636a78f33512 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:26:53.036276 21029 raft_consensus.cc:1261] T b315327141954b53a0f87c2d1dc14b9b P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:26:53.037097 22485 consensus_queue.cc:1035] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:26:53.044095 19097 raft_consensus.cc:1261] T b315327141954b53a0f87c2d1dc14b9b P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:26:53.044864 22485 consensus_queue.cc:1035] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:26:53.048264 20760 raft_consensus.cc:1261] T b315327141954b53a0f87c2d1dc14b9b P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:26:53.049105 18832 raft_consensus.cc:1261] T b315327141954b53a0f87c2d1dc14b9b P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:26:53.050122 21791 consensus_queue.cc:1035] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:26:53.054585 20894 raft_consensus.cc:1203] T 8b8d81c275144f898dcc69a6d98d11e0 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:26:53.074044 22470 ts_tablet_manager.cc:902] T 414e03d597934da3a6a74d786aeab134 P 7ab6d395b30d4afb9821b61c9454430b: Initiating tablet copy from peer fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
I20240502 14:26:53.079988 21029 raft_consensus.cc:1203] T 0d8667d91fd64809b635362c4a8fa33f P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:26:53.093070 18965 raft_consensus.cc:1261] T 96662a3d9e1b4f03b05464ddd06b93d6 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:53.094692 19095 raft_consensus.cc:1261] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:53.096758 22485 consensus_queue.cc:1035] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:53.100804 21791 consensus_queue.cc:1035] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:26:53.131984 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96662a3d9e1b4f03b05464ddd06b93d6. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:26:53.132299 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b315327141954b53a0f87c2d1dc14b9b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:53.133055 18832 consensus_queue.cc:237] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
W20240502 14:26:53.149147 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8b8d81c275144f898dcc69a6d98d11e0. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:53.150403 20893 raft_consensus.cc:1261] T 8b8d81c275144f898dcc69a6d98d11e0 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:53.151010 22500 consensus_queue.cc:1035] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:53.161679 18964 raft_consensus.cc:1261] T 8b8d81c275144f898dcc69a6d98d11e0 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:53.162650 22500 consensus_queue.cc:1035] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:53.169010 18701 raft_consensus.cc:1261] T 8b8d81c275144f898dcc69a6d98d11e0 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:53.171013 22490 consensus_queue.cc:1035] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:26:53.184808 18637 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.68:41463, user_credentials={real_user=slave}} blocked reactor thread for 50840us
I20240502 14:26:53.202522 22485 consensus_queue.cc:1035] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:26:53.214017 22604 tablet_bootstrap.cc:492] T bad03a4ec8494938839d7d6eef8c684e P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:53.214846 22604 tablet_bootstrap.cc:492] T bad03a4ec8494938839d7d6eef8c684e P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap complete.
I20240502 14:26:53.215185 22604 ts_tablet_manager.cc:1367] T bad03a4ec8494938839d7d6eef8c684e P a14c220ad2cc4bf9bd7dad65f4803eb0: Time spent bootstrapping tablet: real 0.280s user 0.072s sys 0.001s
I20240502 14:26:53.216209 22604 raft_consensus.cc:348] T bad03a4ec8494938839d7d6eef8c684e P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:26:53.216521 22604 raft_consensus.cc:729] T bad03a4ec8494938839d7d6eef8c684e P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a14c220ad2cc4bf9bd7dad65f4803eb0, State: Initialized, Role: FOLLOWER
I20240502 14:26:53.216833 22604 consensus_queue.cc:260] T bad03a4ec8494938839d7d6eef8c684e P a14c220ad2cc4bf9bd7dad65f4803eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:26:53.217669 22604 ts_tablet_manager.cc:1398] T bad03a4ec8494938839d7d6eef8c684e P a14c220ad2cc4bf9bd7dad65f4803eb0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:26:53.218843 19117 tablet_copy_service.cc:342] P 007b2e79dc424a6f80141180f072f2d8: Request end of tablet copy session a14c220ad2cc4bf9bd7dad65f4803eb0-bad03a4ec8494938839d7d6eef8c684e received from {username='slave'} at 127.5.107.70:47193
I20240502 14:26:53.219024 19117 tablet_copy_service.cc:433] P 007b2e79dc424a6f80141180f072f2d8: ending tablet copy session a14c220ad2cc4bf9bd7dad65f4803eb0-bad03a4ec8494938839d7d6eef8c684e on tablet bad03a4ec8494938839d7d6eef8c684e with peer a14c220ad2cc4bf9bd7dad65f4803eb0
I20240502 14:26:53.255156 22485 raft_consensus.cc:2932] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:53.258239 18832 raft_consensus.cc:2932] T b315327141954b53a0f87c2d1dc14b9b P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:53.272329 18558 catalog_manager.cc:5484] T b315327141954b53a0f87c2d1dc14b9b P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index 5 to 6, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:53.283063 20760 raft_consensus.cc:2932] T b315327141954b53a0f87c2d1dc14b9b P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:53.291390 21029 raft_consensus.cc:2932] T b315327141954b53a0f87c2d1dc14b9b P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:53.312103 22470 tablet_copy_client.cc:308] T 414e03d597934da3a6a74d786aeab134 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:33009
I20240502 14:26:53.312932 18854 tablet_copy_service.cc:140] P fec04c50dc5b49878ffa3c3873df32e4: Received BeginTabletCopySession request for tablet 414e03d597934da3a6a74d786aeab134 from peer 7ab6d395b30d4afb9821b61c9454430b ({username='slave'} at 127.5.107.71:44135)
I20240502 14:26:53.313159 18854 tablet_copy_service.cc:161] P fec04c50dc5b49878ffa3c3873df32e4: Beginning new tablet copy session on tablet 414e03d597934da3a6a74d786aeab134 from peer 7ab6d395b30d4afb9821b61c9454430b at {username='slave'} at 127.5.107.71:44135: session id = 7ab6d395b30d4afb9821b61c9454430b-414e03d597934da3a6a74d786aeab134
I20240502 14:26:53.317420 18854 tablet_copy_source_session.cc:205] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:53.317035 22490 raft_consensus.cc:2932] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:53.324834 18963 raft_consensus.cc:2932] T 8b8d81c275144f898dcc69a6d98d11e0 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:53.332209 18552 catalog_manager.cc:5484] T 8b8d81c275144f898dcc69a6d98d11e0 P c902805060df4df9b5599d0e94b2dc06 reported cstate change: config changed from index 4 to 5, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:53.332405 18701 raft_consensus.cc:2932] T 8b8d81c275144f898dcc69a6d98d11e0 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:53.349285 22470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 414e03d597934da3a6a74d786aeab134. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:53.353592 20894 raft_consensus.cc:2932] T 8b8d81c275144f898dcc69a6d98d11e0 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:53.404587 20760 raft_consensus.cc:1203] T bad03a4ec8494938839d7d6eef8c684e P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:26:53.406108 22674 ts_tablet_manager.cc:902] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681: Initiating tablet copy from peer fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
I20240502 14:26:53.406920 22470 tablet_copy_client.cc:783] T 414e03d597934da3a6a74d786aeab134 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:53.407218 22470 tablet_copy_client.cc:647] T 414e03d597934da3a6a74d786aeab134 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Starting download of 1 WAL segments...
W20240502 14:26:53.408453 18637 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.67:45409, user_credentials={real_user=slave}} blocked reactor thread for 59626.7us
I20240502 14:26:53.410871 21791 raft_consensus.cc:2932] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:53.416769 18963 raft_consensus.cc:2932] T 96662a3d9e1b4f03b05464ddd06b93d6 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:53.422633 19093 raft_consensus.cc:2932] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:53.429322 18556 catalog_manager.cc:5484] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: config changed from index -1 to 4, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:53.436333 19097 raft_consensus.cc:2932] T b315327141954b53a0f87c2d1dc14b9b P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:53.437942 22470 tablet_copy_client.cc:523] T 414e03d597934da3a6a74d786aeab134 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:53.441437 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:53.442593 22470 tablet_bootstrap.cc:492] T 414e03d597934da3a6a74d786aeab134 P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap starting.
I20240502 14:26:53.449339 19076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:26:53.456929 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:26:53.523536 22674 tablet_copy_client.cc:308] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:33009
I20240502 14:26:53.542093 18854 tablet_copy_service.cc:140] P fec04c50dc5b49878ffa3c3873df32e4: Received BeginTabletCopySession request for tablet 522014370f0a4cc3bd31aa00d6a05c5e from peer a834edffbcb2469aae598b069aab0681 ({username='slave'} at 127.5.107.69:37387)
I20240502 14:26:53.542266 18854 tablet_copy_service.cc:161] P fec04c50dc5b49878ffa3c3873df32e4: Beginning new tablet copy session on tablet 522014370f0a4cc3bd31aa00d6a05c5e from peer a834edffbcb2469aae598b069aab0681 at {username='slave'} at 127.5.107.69:37387: session id = a834edffbcb2469aae598b069aab0681-522014370f0a4cc3bd31aa00d6a05c5e
I20240502 14:26:53.545500 18854 tablet_copy_source_session.cc:205] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:53.550693 22674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 522014370f0a4cc3bd31aa00d6a05c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:53.558796 22674 tablet_copy_client.cc:783] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:53.559065 22674 tablet_copy_client.cc:647] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:53.565361 22674 tablet_copy_client.cc:523] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:53.566398 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:53.568945 22674 tablet_bootstrap.cc:492] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681: Bootstrap starting.
I20240502 14:26:53.614177 22735 ts_tablet_manager.cc:902] T 96662a3d9e1b4f03b05464ddd06b93d6 P 7ab6d395b30d4afb9821b61c9454430b: Initiating tablet copy from peer 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:26:53.615963 22736 ts_tablet_manager.cc:902] T b315327141954b53a0f87c2d1dc14b9b P 7ab6d395b30d4afb9821b61c9454430b: Initiating tablet copy from peer 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:26:53.631851 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:26:53.638399 21007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:53.642561 22721 ts_tablet_manager.cc:902] T 8b8d81c275144f898dcc69a6d98d11e0 P ff6acba046404ef0a6f41ad5e09d852f: Initiating tablet copy from peer fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
I20240502 14:26:53.655010 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 14:26:53.672899 18898 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.65:36749, user_credentials={real_user=slave}} blocked reactor thread for 149194us
I20240502 14:26:53.714848 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:53.733582 22721 tablet_copy_client.cc:308] T 8b8d81c275144f898dcc69a6d98d11e0 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:33009
I20240502 14:26:53.734725 18854 tablet_copy_service.cc:140] P fec04c50dc5b49878ffa3c3873df32e4: Received BeginTabletCopySession request for tablet 8b8d81c275144f898dcc69a6d98d11e0 from peer ff6acba046404ef0a6f41ad5e09d852f ({username='slave'} at 127.5.107.72:58697)
I20240502 14:26:53.734913 18854 tablet_copy_service.cc:161] P fec04c50dc5b49878ffa3c3873df32e4: Beginning new tablet copy session on tablet 8b8d81c275144f898dcc69a6d98d11e0 from peer ff6acba046404ef0a6f41ad5e09d852f at {username='slave'} at 127.5.107.72:58697: session id = ff6acba046404ef0a6f41ad5e09d852f-8b8d81c275144f898dcc69a6d98d11e0
I20240502 14:26:53.740857 18854 tablet_copy_source_session.cc:205] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:53.746997 22721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b8d81c275144f898dcc69a6d98d11e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:53.761309 22721 tablet_copy_client.cc:783] T 8b8d81c275144f898dcc69a6d98d11e0 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:53.761636 22721 tablet_copy_client.cc:647] T 8b8d81c275144f898dcc69a6d98d11e0 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:53.763574 22721 tablet_copy_client.cc:523] T 8b8d81c275144f898dcc69a6d98d11e0 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:53.767131 22721 tablet_bootstrap.cc:492] T 8b8d81c275144f898dcc69a6d98d11e0 P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap starting.
I20240502 14:26:53.782804 22735 tablet_copy_client.cc:308] T 96662a3d9e1b4f03b05464ddd06b93d6 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.65:36749
I20240502 14:26:53.783164 22736 tablet_copy_client.cc:308] T b315327141954b53a0f87c2d1dc14b9b P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.65:36749
I20240502 14:26:53.814694 22674 tablet_bootstrap.cc:492] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:53.815851 22674 tablet_bootstrap.cc:492] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681: Bootstrap complete.
I20240502 14:26:53.816913 22470 tablet_bootstrap.cc:492] T 414e03d597934da3a6a74d786aeab134 P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:53.817428 22470 tablet_bootstrap.cc:492] T 414e03d597934da3a6a74d786aeab134 P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap complete.
I20240502 14:26:53.817716 22470 ts_tablet_manager.cc:1367] T 414e03d597934da3a6a74d786aeab134 P 7ab6d395b30d4afb9821b61c9454430b: Time spent bootstrapping tablet: real 0.375s user 0.058s sys 0.026s
I20240502 14:26:53.817644 18833 consensus_queue.cc:237] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:53.818506 22470 raft_consensus.cc:348] T 414e03d597934da3a6a74d786aeab134 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:53.818781 22470 raft_consensus.cc:729] T 414e03d597934da3a6a74d786aeab134 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab6d395b30d4afb9821b61c9454430b, State: Initialized, Role: FOLLOWER
I20240502 14:26:53.819051 22470 consensus_queue.cc:260] T 414e03d597934da3a6a74d786aeab134 P 7ab6d395b30d4afb9821b61c9454430b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:53.819543 22470 ts_tablet_manager.cc:1398] T 414e03d597934da3a6a74d786aeab134 P 7ab6d395b30d4afb9821b61c9454430b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:26:53.827932 18854 tablet_copy_service.cc:342] P fec04c50dc5b49878ffa3c3873df32e4: Request end of tablet copy session 7ab6d395b30d4afb9821b61c9454430b-414e03d597934da3a6a74d786aeab134 received from {username='slave'} at 127.5.107.71:44135
I20240502 14:26:53.828079 18854 tablet_copy_service.cc:433] P fec04c50dc5b49878ffa3c3873df32e4: ending tablet copy session 7ab6d395b30d4afb9821b61c9454430b-414e03d597934da3a6a74d786aeab134 on tablet 414e03d597934da3a6a74d786aeab134 with peer 7ab6d395b30d4afb9821b61c9454430b
I20240502 14:26:53.834259 22674 ts_tablet_manager.cc:1367] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681: Time spent bootstrapping tablet: real 0.247s user 0.058s sys 0.044s
I20240502 14:26:53.835045 22674 raft_consensus.cc:348] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:53.835335 22674 raft_consensus.cc:729] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a834edffbcb2469aae598b069aab0681, State: Initialized, Role: FOLLOWER
I20240502 14:26:53.835575 22674 consensus_queue.cc:260] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:53.837472 18701 raft_consensus.cc:1261] T 2c2afaffde96462eacd0d32bbdec23b1 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:26:53.837519 18769 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c2afaffde96462eacd0d32bbdec23b1. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:53.838333 18965 raft_consensus.cc:1261] T 2c2afaffde96462eacd0d32bbdec23b1 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:53.838245 22490 consensus_queue.cc:1035] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:53.838873 22490 consensus_queue.cc:1035] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:53.848393 18721 tablet_copy_service.cc:140] P 2db304b538e14857b7b1cc272d0db94b: Received BeginTabletCopySession request for tablet 96662a3d9e1b4f03b05464ddd06b93d6 from peer 7ab6d395b30d4afb9821b61c9454430b ({username='slave'} at 127.5.107.71:40777)
I20240502 14:26:53.848455 18720 tablet_copy_service.cc:140] P 2db304b538e14857b7b1cc272d0db94b: Received BeginTabletCopySession request for tablet b315327141954b53a0f87c2d1dc14b9b from peer 7ab6d395b30d4afb9821b61c9454430b ({username='slave'} at 127.5.107.71:40777)
I20240502 14:26:53.848583 18721 tablet_copy_service.cc:161] P 2db304b538e14857b7b1cc272d0db94b: Beginning new tablet copy session on tablet 96662a3d9e1b4f03b05464ddd06b93d6 from peer 7ab6d395b30d4afb9821b61c9454430b at {username='slave'} at 127.5.107.71:40777: session id = 7ab6d395b30d4afb9821b61c9454430b-96662a3d9e1b4f03b05464ddd06b93d6
I20240502 14:26:53.848783 18720 tablet_copy_service.cc:161] P 2db304b538e14857b7b1cc272d0db94b: Beginning new tablet copy session on tablet b315327141954b53a0f87c2d1dc14b9b from peer 7ab6d395b30d4afb9821b61c9454430b at {username='slave'} at 127.5.107.71:40777: session id = 7ab6d395b30d4afb9821b61c9454430b-b315327141954b53a0f87c2d1dc14b9b
I20240502 14:26:53.852607 18721 tablet_copy_source_session.cc:205] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:53.854138 22735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96662a3d9e1b4f03b05464ddd06b93d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:53.855850 18720 tablet_copy_source_session.cc:205] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:53.857775 22736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b315327141954b53a0f87c2d1dc14b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:53.866595 22736 tablet_copy_client.cc:783] T b315327141954b53a0f87c2d1dc14b9b P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:53.866945 22736 tablet_copy_client.cc:647] T b315327141954b53a0f87c2d1dc14b9b P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:53.868209 22735 tablet_copy_client.cc:783] T 96662a3d9e1b4f03b05464ddd06b93d6 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:53.868559 22735 tablet_copy_client.cc:647] T 96662a3d9e1b4f03b05464ddd06b93d6 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:53.868324 18701 consensus_queue.cc:237] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:26:53.870103 22736 tablet_copy_client.cc:523] T b315327141954b53a0f87c2d1dc14b9b P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:53.870978 22735 tablet_copy_client.cc:523] T 96662a3d9e1b4f03b05464ddd06b93d6 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:53.882670 19097 consensus_queue.cc:237] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:53.886153 22736 tablet_bootstrap.cc:492] T b315327141954b53a0f87c2d1dc14b9b P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap starting.
I20240502 14:26:53.886955 18965 raft_consensus.cc:2932] T 2c2afaffde96462eacd0d32bbdec23b1 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:53.896365 18553 catalog_manager.cc:5484] T 2c2afaffde96462eacd0d32bbdec23b1 P c902805060df4df9b5599d0e94b2dc06 reported cstate change: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:53.900185 22735 tablet_bootstrap.cc:492] T 96662a3d9e1b4f03b05464ddd06b93d6 P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap starting.
I20240502 14:26:53.908195 22776 raft_consensus.cc:2932] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:53.909682 22721 tablet_bootstrap.cc:492] T 8b8d81c275144f898dcc69a6d98d11e0 P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:53.910152 22721 tablet_bootstrap.cc:492] T 8b8d81c275144f898dcc69a6d98d11e0 P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap complete.
I20240502 14:26:53.910467 22721 ts_tablet_manager.cc:1367] T 8b8d81c275144f898dcc69a6d98d11e0 P ff6acba046404ef0a6f41ad5e09d852f: Time spent bootstrapping tablet: real 0.143s user 0.061s sys 0.003s
I20240502 14:26:53.911417 22721 raft_consensus.cc:348] T 8b8d81c275144f898dcc69a6d98d11e0 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:26:53.911765 22721 raft_consensus.cc:729] T 8b8d81c275144f898dcc69a6d98d11e0 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6acba046404ef0a6f41ad5e09d852f, State: Initialized, Role: FOLLOWER
I20240502 14:26:53.912011 22721 consensus_queue.cc:260] T 8b8d81c275144f898dcc69a6d98d11e0 P ff6acba046404ef0a6f41ad5e09d852f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
W20240502 14:26:53.914347 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ffb6e582ef04fb38dd273c13febac58. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:53.916182 22674 ts_tablet_manager.cc:1398] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681: Time spent starting tablet: real 0.082s user 0.017s sys 0.004s
I20240502 14:26:53.916914 18854 tablet_copy_service.cc:342] P fec04c50dc5b49878ffa3c3873df32e4: Request end of tablet copy session a834edffbcb2469aae598b069aab0681-522014370f0a4cc3bd31aa00d6a05c5e received from {username='slave'} at 127.5.107.69:37387
I20240502 14:26:53.917030 18854 tablet_copy_service.cc:433] P fec04c50dc5b49878ffa3c3873df32e4: ending tablet copy session a834edffbcb2469aae598b069aab0681-522014370f0a4cc3bd31aa00d6a05c5e on tablet 522014370f0a4cc3bd31aa00d6a05c5e with peer a834edffbcb2469aae598b069aab0681
W20240502 14:26:53.927541 19032 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 097eb705c9e24740a5be61bf3354f3d8. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:53.928264 18833 raft_consensus.cc:1261] T 097eb705c9e24740a5be61bf3354f3d8 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:53.928769 22519 consensus_queue.cc:1035] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:53.934657 18700 raft_consensus.cc:2932] T 2c2afaffde96462eacd0d32bbdec23b1 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:53.945369 18700 raft_consensus.cc:1261] T 097eb705c9e24740a5be61bf3354f3d8 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:53.946345 22519 consensus_queue.cc:1035] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:53.965417 18965 raft_consensus.cc:1261] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:53.965854 21029 raft_consensus.cc:1261] T 0ffb6e582ef04fb38dd273c13febac58 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:53.966223 22650 consensus_queue.cc:1035] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:53.963517 22786 raft_consensus.cc:2932] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:53.966147 18699 raft_consensus.cc:2932] T 097eb705c9e24740a5be61bf3354f3d8 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:53.966802 22650 consensus_queue.cc:1035] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:53.970577 18833 raft_consensus.cc:1261] T 0ffb6e582ef04fb38dd273c13febac58 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:53.981139 21791 consensus_queue.cc:1035] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 14:26:53.989825 18833 raft_consensus.cc:2932] T 097eb705c9e24740a5be61bf3354f3d8 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:53.992094 18553 catalog_manager.cc:5484] T 097eb705c9e24740a5be61bf3354f3d8 P 2db304b538e14857b7b1cc272d0db94b reported cstate change: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:53.999779 22650 raft_consensus.cc:2932] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:54.004098 18832 raft_consensus.cc:2932] T 0ffb6e582ef04fb38dd273c13febac58 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:54.010684 18556 catalog_manager.cc:5484] T 0ffb6e582ef04fb38dd273c13febac58 P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index 4 to 5, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:54.014050 22736 tablet_bootstrap.cc:492] T b315327141954b53a0f87c2d1dc14b9b P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:54.014559 22736 tablet_bootstrap.cc:492] T b315327141954b53a0f87c2d1dc14b9b P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap complete.
I20240502 14:26:54.014907 22736 ts_tablet_manager.cc:1367] T b315327141954b53a0f87c2d1dc14b9b P 7ab6d395b30d4afb9821b61c9454430b: Time spent bootstrapping tablet: real 0.129s user 0.048s sys 0.005s
I20240502 14:26:54.018241 21029 raft_consensus.cc:2932] T 0ffb6e582ef04fb38dd273c13febac58 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:54.018618 18965 raft_consensus.cc:2932] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:54.030818 22721 ts_tablet_manager.cc:1398] T 8b8d81c275144f898dcc69a6d98d11e0 P ff6acba046404ef0a6f41ad5e09d852f: Time spent starting tablet: real 0.120s user 0.028s sys 0.021s
I20240502 14:26:54.031559 18854 tablet_copy_service.cc:342] P fec04c50dc5b49878ffa3c3873df32e4: Request end of tablet copy session ff6acba046404ef0a6f41ad5e09d852f-8b8d81c275144f898dcc69a6d98d11e0 received from {username='slave'} at 127.5.107.72:58697
I20240502 14:26:54.031664 18854 tablet_copy_service.cc:433] P fec04c50dc5b49878ffa3c3873df32e4: ending tablet copy session ff6acba046404ef0a6f41ad5e09d852f-8b8d81c275144f898dcc69a6d98d11e0 on tablet 8b8d81c275144f898dcc69a6d98d11e0 with peer ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:26:54.034382 22736 raft_consensus.cc:348] T b315327141954b53a0f87c2d1dc14b9b P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:54.034777 22736 raft_consensus.cc:729] T b315327141954b53a0f87c2d1dc14b9b P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab6d395b30d4afb9821b61c9454430b, State: Initialized, Role: FOLLOWER
I20240502 14:26:54.035072 22736 consensus_queue.cc:260] T b315327141954b53a0f87c2d1dc14b9b P 7ab6d395b30d4afb9821b61c9454430b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:54.035833 22736 ts_tablet_manager.cc:1398] T b315327141954b53a0f87c2d1dc14b9b P 7ab6d395b30d4afb9821b61c9454430b: Time spent starting tablet: real 0.021s user 0.001s sys 0.000s
I20240502 14:26:54.038091 22735 tablet_bootstrap.cc:492] T 96662a3d9e1b4f03b05464ddd06b93d6 P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:54.038493 22735 tablet_bootstrap.cc:492] T 96662a3d9e1b4f03b05464ddd06b93d6 P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap complete.
I20240502 14:26:54.038755 22735 ts_tablet_manager.cc:1367] T 96662a3d9e1b4f03b05464ddd06b93d6 P 7ab6d395b30d4afb9821b61c9454430b: Time spent bootstrapping tablet: real 0.139s user 0.040s sys 0.008s
I20240502 14:26:54.039551 22735 raft_consensus.cc:348] T 96662a3d9e1b4f03b05464ddd06b93d6 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:54.039844 22735 raft_consensus.cc:729] T 96662a3d9e1b4f03b05464ddd06b93d6 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab6d395b30d4afb9821b61c9454430b, State: Initialized, Role: FOLLOWER
I20240502 14:26:54.040118 22735 consensus_queue.cc:260] T 96662a3d9e1b4f03b05464ddd06b93d6 P 7ab6d395b30d4afb9821b61c9454430b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:54.040673 22735 ts_tablet_manager.cc:1398] T 96662a3d9e1b4f03b05464ddd06b93d6 P 7ab6d395b30d4afb9821b61c9454430b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:26:54.043807 18721 tablet_copy_service.cc:342] P 2db304b538e14857b7b1cc272d0db94b: Request end of tablet copy session 7ab6d395b30d4afb9821b61c9454430b-b315327141954b53a0f87c2d1dc14b9b received from {username='slave'} at 127.5.107.71:40777
I20240502 14:26:54.043965 18721 tablet_copy_service.cc:433] P 2db304b538e14857b7b1cc272d0db94b: ending tablet copy session 7ab6d395b30d4afb9821b61c9454430b-b315327141954b53a0f87c2d1dc14b9b on tablet b315327141954b53a0f87c2d1dc14b9b with peer 7ab6d395b30d4afb9821b61c9454430b
I20240502 14:26:54.059072 18721 tablet_copy_service.cc:342] P 2db304b538e14857b7b1cc272d0db94b: Request end of tablet copy session 7ab6d395b30d4afb9821b61c9454430b-96662a3d9e1b4f03b05464ddd06b93d6 received from {username='slave'} at 127.5.107.71:40777
I20240502 14:26:54.059212 18721 tablet_copy_service.cc:433] P 2db304b538e14857b7b1cc272d0db94b: ending tablet copy session 7ab6d395b30d4afb9821b61c9454430b-96662a3d9e1b4f03b05464ddd06b93d6 on tablet 96662a3d9e1b4f03b05464ddd06b93d6 with peer 7ab6d395b30d4afb9821b61c9454430b
I20240502 14:26:54.063300 21029 raft_consensus.cc:1203] T 8b8d81c275144f898dcc69a6d98d11e0 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:26:54.126196 20894 raft_consensus.cc:1203] T 96662a3d9e1b4f03b05464ddd06b93d6 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:26:54.261893 20894 raft_consensus.cc:1203] T 414e03d597934da3a6a74d786aeab134 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:26:54.331215 20626 raft_consensus.cc:1203] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:26:54.344511 22861 ts_tablet_manager.cc:902] T 0ffb6e582ef04fb38dd273c13febac58 P a14c220ad2cc4bf9bd7dad65f4803eb0: Initiating tablet copy from peer 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:26:54.359086 19097 consensus_queue.cc:237] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:54.370420 20626 raft_consensus.cc:1261] T 8937c809610542d88fdde3daf002c890 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:26:54.371482 21029 raft_consensus.cc:1261] T 8937c809610542d88fdde3daf002c890 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:26:54.372041 22786 consensus_queue.cc:1035] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:26:54.373560 22786 consensus_queue.cc:1035] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:26:54.387670 18965 raft_consensus.cc:1261] T 8937c809610542d88fdde3daf002c890 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:26:54.388410 22792 consensus_queue.cc:1035] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:26:54.400173 18833 raft_consensus.cc:1261] T 8937c809610542d88fdde3daf002c890 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:26:54.400868 22792 consensus_queue.cc:1035] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 14:26:54.410368 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8937c809610542d88fdde3daf002c890. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:54.414628 22861 tablet_copy_client.cc:308] T 0ffb6e582ef04fb38dd273c13febac58 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.65:36749
I20240502 14:26:54.415921 18721 tablet_copy_service.cc:140] P 2db304b538e14857b7b1cc272d0db94b: Received BeginTabletCopySession request for tablet 0ffb6e582ef04fb38dd273c13febac58 from peer a14c220ad2cc4bf9bd7dad65f4803eb0 ({username='slave'} at 127.5.107.70:48397)
I20240502 14:26:54.416118 18721 tablet_copy_service.cc:161] P 2db304b538e14857b7b1cc272d0db94b: Beginning new tablet copy session on tablet 0ffb6e582ef04fb38dd273c13febac58 from peer a14c220ad2cc4bf9bd7dad65f4803eb0 at {username='slave'} at 127.5.107.70:48397: session id = a14c220ad2cc4bf9bd7dad65f4803eb0-0ffb6e582ef04fb38dd273c13febac58
I20240502 14:26:54.422020 22674 ts_tablet_manager.cc:902] T 2c2afaffde96462eacd0d32bbdec23b1 P a834edffbcb2469aae598b069aab0681: Initiating tablet copy from peer fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
I20240502 14:26:54.422371 22792 raft_consensus.cc:2932] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:54.425298 18721 tablet_copy_source_session.cc:205] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:54.427124 22861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ffb6e582ef04fb38dd273c13febac58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:54.440182 22861 tablet_copy_client.cc:783] T 0ffb6e582ef04fb38dd273c13febac58 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:54.440448 22861 tablet_copy_client.cc:647] T 0ffb6e582ef04fb38dd273c13febac58 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:54.443169 18833 raft_consensus.cc:2932] T 8937c809610542d88fdde3daf002c890 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:54.444954 19093 consensus_queue.cc:237] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:26:54.457679 22861 tablet_copy_client.cc:523] T 0ffb6e582ef04fb38dd273c13febac58 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:54.457340 20626 raft_consensus.cc:2932] T 8937c809610542d88fdde3daf002c890 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:54.460438 18556 catalog_manager.cc:5484] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: config changed from index 5 to 6, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:54.463920 22861 tablet_bootstrap.cc:492] T 0ffb6e582ef04fb38dd273c13febac58 P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap starting.
I20240502 14:26:54.468691 21029 raft_consensus.cc:2932] T 8937c809610542d88fdde3daf002c890 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:54.476759 22886 ts_tablet_manager.cc:902] T 097eb705c9e24740a5be61bf3354f3d8 P a834edffbcb2469aae598b069aab0681: Initiating tablet copy from peer 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
W20240502 14:26:54.482542 19031 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b15d11209e45473e863766e5cd6e822f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:54.485954 18701 raft_consensus.cc:1261] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:54.486972 18964 raft_consensus.cc:2932] T 8937c809610542d88fdde3daf002c890 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:54.487134 22792 consensus_queue.cc:1035] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:54.495898 18964 raft_consensus.cc:1261] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:54.499625 22786 consensus_queue.cc:1035] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:54.506245 20894 raft_consensus.cc:1203] T b315327141954b53a0f87c2d1dc14b9b P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 14:26:54.558481 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:26:54.582523 22674 tablet_copy_client.cc:308] T 2c2afaffde96462eacd0d32bbdec23b1 P a834edffbcb2469aae598b069aab0681: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:33009
I20240502 14:26:54.583278 18854 tablet_copy_service.cc:140] P fec04c50dc5b49878ffa3c3873df32e4: Received BeginTabletCopySession request for tablet 2c2afaffde96462eacd0d32bbdec23b1 from peer a834edffbcb2469aae598b069aab0681 ({username='slave'} at 127.5.107.69:37387)
I20240502 14:26:54.583441 18854 tablet_copy_service.cc:161] P fec04c50dc5b49878ffa3c3873df32e4: Beginning new tablet copy session on tablet 2c2afaffde96462eacd0d32bbdec23b1 from peer a834edffbcb2469aae598b069aab0681 at {username='slave'} at 127.5.107.69:37387: session id = a834edffbcb2469aae598b069aab0681-2c2afaffde96462eacd0d32bbdec23b1
I20240502 14:26:54.586380 18854 tablet_copy_source_session.cc:205] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:54.589799 22674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c2afaffde96462eacd0d32bbdec23b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:54.596400 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:54.601444 22786 raft_consensus.cc:2932] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:54.606144 18965 raft_consensus.cc:2932] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:54.613101 22674 tablet_copy_client.cc:783] T 2c2afaffde96462eacd0d32bbdec23b1 P a834edffbcb2469aae598b069aab0681: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:54.613333 22674 tablet_copy_client.cc:647] T 2c2afaffde96462eacd0d32bbdec23b1 P a834edffbcb2469aae598b069aab0681: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:54.616138 22674 tablet_copy_client.cc:523] T 2c2afaffde96462eacd0d32bbdec23b1 P a834edffbcb2469aae598b069aab0681: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:54.620604 22674 tablet_bootstrap.cc:492] T 2c2afaffde96462eacd0d32bbdec23b1 P a834edffbcb2469aae598b069aab0681: Bootstrap starting.
I20240502 14:26:54.625712 18554 catalog_manager.cc:5484] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 reported cstate change: config changed from index -1 to 4, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:54.626233 18701 raft_consensus.cc:2932] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:54.658954 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:54.683873 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:26:54.684643 19076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:26:54.713899 22886 tablet_copy_client.cc:308] T 097eb705c9e24740a5be61bf3354f3d8 P a834edffbcb2469aae598b069aab0681: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.68:41463
I20240502 14:26:54.717464 19117 tablet_copy_service.cc:140] P 007b2e79dc424a6f80141180f072f2d8: Received BeginTabletCopySession request for tablet 097eb705c9e24740a5be61bf3354f3d8 from peer a834edffbcb2469aae598b069aab0681 ({username='slave'} at 127.5.107.69:37525)
I20240502 14:26:54.717660 19117 tablet_copy_service.cc:161] P 007b2e79dc424a6f80141180f072f2d8: Beginning new tablet copy session on tablet 097eb705c9e24740a5be61bf3354f3d8 from peer a834edffbcb2469aae598b069aab0681 at {username='slave'} at 127.5.107.69:37525: session id = a834edffbcb2469aae598b069aab0681-097eb705c9e24740a5be61bf3354f3d8
I20240502 14:26:54.720734 19117 tablet_copy_source_session.cc:205] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:54.722123 22886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 097eb705c9e24740a5be61bf3354f3d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:54.722170 22861 tablet_bootstrap.cc:492] T 0ffb6e582ef04fb38dd273c13febac58 P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:54.722625 22861 tablet_bootstrap.cc:492] T 0ffb6e582ef04fb38dd273c13febac58 P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap complete.
I20240502 14:26:54.722940 22861 ts_tablet_manager.cc:1367] T 0ffb6e582ef04fb38dd273c13febac58 P a14c220ad2cc4bf9bd7dad65f4803eb0: Time spent bootstrapping tablet: real 0.259s user 0.059s sys 0.009s
I20240502 14:26:54.723745 22861 raft_consensus.cc:348] T 0ffb6e582ef04fb38dd273c13febac58 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:26:54.724035 22861 raft_consensus.cc:729] T 0ffb6e582ef04fb38dd273c13febac58 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a14c220ad2cc4bf9bd7dad65f4803eb0, State: Initialized, Role: FOLLOWER
I20240502 14:26:54.724316 22861 consensus_queue.cc:260] T 0ffb6e582ef04fb38dd273c13febac58 P a14c220ad2cc4bf9bd7dad65f4803eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:26:54.732285 22861 ts_tablet_manager.cc:1398] T 0ffb6e582ef04fb38dd273c13febac58 P a14c220ad2cc4bf9bd7dad65f4803eb0: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s
I20240502 14:26:54.732995 18721 tablet_copy_service.cc:342] P 2db304b538e14857b7b1cc272d0db94b: Request end of tablet copy session a14c220ad2cc4bf9bd7dad65f4803eb0-0ffb6e582ef04fb38dd273c13febac58 received from {username='slave'} at 127.5.107.70:48397
I20240502 14:26:54.733109 18721 tablet_copy_service.cc:433] P 2db304b538e14857b7b1cc272d0db94b: ending tablet copy session a14c220ad2cc4bf9bd7dad65f4803eb0-0ffb6e582ef04fb38dd273c13febac58 on tablet 0ffb6e582ef04fb38dd273c13febac58 with peer a14c220ad2cc4bf9bd7dad65f4803eb0
I20240502 14:26:54.745786 22886 tablet_copy_client.cc:783] T 097eb705c9e24740a5be61bf3354f3d8 P a834edffbcb2469aae598b069aab0681: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:54.746065 22886 tablet_copy_client.cc:647] T 097eb705c9e24740a5be61bf3354f3d8 P a834edffbcb2469aae598b069aab0681: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:54.759737 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:54.778499 21007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:54.796281 22886 tablet_copy_client.cc:523] T 097eb705c9e24740a5be61bf3354f3d8 P a834edffbcb2469aae598b069aab0681: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:54.803264 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:54.819234 22886 tablet_bootstrap.cc:492] T 097eb705c9e24740a5be61bf3354f3d8 P a834edffbcb2469aae598b069aab0681: Bootstrap starting.
I20240502 14:26:54.819350 18964 consensus_queue.cc:237] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:54.838716 18833 raft_consensus.cc:1261] T b13bcd6e693f4c7687a1aabdafbb83c6 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:54.839154 18701 raft_consensus.cc:1261] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:54.839359 21881 consensus_queue.cc:1035] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:54.839834 22912 consensus_queue.cc:1035] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:54.881067 18701 consensus_queue.cc:237] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:26:54.887794 22912 raft_consensus.cc:2932] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:54.891583 20760 raft_consensus.cc:1203] T 0ffb6e582ef04fb38dd273c13febac58 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:26:54.892702 18833 raft_consensus.cc:2932] T b13bcd6e693f4c7687a1aabdafbb83c6 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:54.893688 18699 raft_consensus.cc:2932] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:54.901650 18556 catalog_manager.cc:5484] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b reported cstate change: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
W20240502 14:26:54.902277 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96662a3d9e1b4f03b05464ddd06b93d6. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:54.903331 19093 raft_consensus.cc:1261] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:54.906960 22793 consensus_queue.cc:1035] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:26:54.908608 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b13bcd6e693f4c7687a1aabdafbb83c6. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:54.914407 20894 raft_consensus.cc:1261] T 96662a3d9e1b4f03b05464ddd06b93d6 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:54.917469 21791 consensus_queue.cc:1035] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:54.924305 18964 raft_consensus.cc:1261] T 96662a3d9e1b4f03b05464ddd06b93d6 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:54.924901 21791 consensus_queue.cc:1035] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:54.927654 21791 raft_consensus.cc:2932] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:54.929827 20894 raft_consensus.cc:2932] T 96662a3d9e1b4f03b05464ddd06b93d6 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:54.945693 18557 catalog_manager.cc:5484] T 96662a3d9e1b4f03b05464ddd06b93d6 P 7ab6d395b30d4afb9821b61c9454430b reported cstate change: config changed from index 4 to 5, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:54.947561 19093 raft_consensus.cc:2932] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:54.954726 22674 tablet_bootstrap.cc:492] T 2c2afaffde96462eacd0d32bbdec23b1 P a834edffbcb2469aae598b069aab0681: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:54.958647 22674 tablet_bootstrap.cc:492] T 2c2afaffde96462eacd0d32bbdec23b1 P a834edffbcb2469aae598b069aab0681: Bootstrap complete.
I20240502 14:26:54.960683 22674 ts_tablet_manager.cc:1367] T 2c2afaffde96462eacd0d32bbdec23b1 P a834edffbcb2469aae598b069aab0681: Time spent bootstrapping tablet: real 0.338s user 0.051s sys 0.002s
I20240502 14:26:54.963984 22674 raft_consensus.cc:348] T 2c2afaffde96462eacd0d32bbdec23b1 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:54.964210 22674 raft_consensus.cc:729] T 2c2afaffde96462eacd0d32bbdec23b1 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a834edffbcb2469aae598b069aab0681, State: Initialized, Role: FOLLOWER
I20240502 14:26:54.964458 22674 consensus_queue.cc:260] T 2c2afaffde96462eacd0d32bbdec23b1 P a834edffbcb2469aae598b069aab0681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:54.965529 22988 ts_tablet_manager.cc:902] T 8937c809610542d88fdde3daf002c890 P 7ab6d395b30d4afb9821b61c9454430b: Initiating tablet copy from peer 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:26:54.978094 18964 raft_consensus.cc:2932] T 96662a3d9e1b4f03b05464ddd06b93d6 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:54.983090 22984 ts_tablet_manager.cc:902] T b15d11209e45473e863766e5cd6e822f P ff6acba046404ef0a6f41ad5e09d852f: Initiating tablet copy from peer 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:26:55.011322 22674 ts_tablet_manager.cc:1398] T 2c2afaffde96462eacd0d32bbdec23b1 P a834edffbcb2469aae598b069aab0681: Time spent starting tablet: real 0.051s user 0.003s sys 0.000s
I20240502 14:26:55.012871 18854 tablet_copy_service.cc:342] P fec04c50dc5b49878ffa3c3873df32e4: Request end of tablet copy session a834edffbcb2469aae598b069aab0681-2c2afaffde96462eacd0d32bbdec23b1 received from {username='slave'} at 127.5.107.69:37387
I20240502 14:26:55.013047 18854 tablet_copy_service.cc:433] P fec04c50dc5b49878ffa3c3873df32e4: ending tablet copy session a834edffbcb2469aae598b069aab0681-2c2afaffde96462eacd0d32bbdec23b1 on tablet 2c2afaffde96462eacd0d32bbdec23b1 with peer a834edffbcb2469aae598b069aab0681
I20240502 14:26:55.014446 19076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:26:55.020377 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:26:55.048192 22984 tablet_copy_client.cc:308] T b15d11209e45473e863766e5cd6e822f P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.68:41463
I20240502 14:26:55.048929 19117 tablet_copy_service.cc:140] P 007b2e79dc424a6f80141180f072f2d8: Received BeginTabletCopySession request for tablet b15d11209e45473e863766e5cd6e822f from peer ff6acba046404ef0a6f41ad5e09d852f ({username='slave'} at 127.5.107.72:33829)
I20240502 14:26:55.049088 19117 tablet_copy_service.cc:161] P 007b2e79dc424a6f80141180f072f2d8: Beginning new tablet copy session on tablet b15d11209e45473e863766e5cd6e822f from peer ff6acba046404ef0a6f41ad5e09d852f at {username='slave'} at 127.5.107.72:33829: session id = ff6acba046404ef0a6f41ad5e09d852f-b15d11209e45473e863766e5cd6e822f
I20240502 14:26:55.052224 19117 tablet_copy_source_session.cc:205] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:55.053758 22984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b15d11209e45473e863766e5cd6e822f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:55.054239 22886 tablet_bootstrap.cc:492] T 097eb705c9e24740a5be61bf3354f3d8 P a834edffbcb2469aae598b069aab0681: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:55.054795 22886 tablet_bootstrap.cc:492] T 097eb705c9e24740a5be61bf3354f3d8 P a834edffbcb2469aae598b069aab0681: Bootstrap complete.
I20240502 14:26:55.058858 22886 ts_tablet_manager.cc:1367] T 097eb705c9e24740a5be61bf3354f3d8 P a834edffbcb2469aae598b069aab0681: Time spent bootstrapping tablet: real 0.240s user 0.045s sys 0.007s
I20240502 14:26:55.059811 22886 raft_consensus.cc:348] T 097eb705c9e24740a5be61bf3354f3d8 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:55.060088 22886 raft_consensus.cc:729] T 097eb705c9e24740a5be61bf3354f3d8 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a834edffbcb2469aae598b069aab0681, State: Initialized, Role: FOLLOWER
I20240502 14:26:55.061125 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:55.062829 22886 consensus_queue.cc:260] T 097eb705c9e24740a5be61bf3354f3d8 P a834edffbcb2469aae598b069aab0681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:55.063566 22886 ts_tablet_manager.cc:1398] T 097eb705c9e24740a5be61bf3354f3d8 P a834edffbcb2469aae598b069aab0681: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 14:26:55.069622 19117 tablet_copy_service.cc:342] P 007b2e79dc424a6f80141180f072f2d8: Request end of tablet copy session a834edffbcb2469aae598b069aab0681-097eb705c9e24740a5be61bf3354f3d8 received from {username='slave'} at 127.5.107.69:37525
I20240502 14:26:55.069833 19117 tablet_copy_service.cc:433] P 007b2e79dc424a6f80141180f072f2d8: ending tablet copy session a834edffbcb2469aae598b069aab0681-097eb705c9e24740a5be61bf3354f3d8 on tablet 097eb705c9e24740a5be61bf3354f3d8 with peer a834edffbcb2469aae598b069aab0681
I20240502 14:26:55.075630 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:55.090394 22988 tablet_copy_client.cc:308] T 8937c809610542d88fdde3daf002c890 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.68:41463
I20240502 14:26:55.103602 22984 tablet_copy_client.cc:783] T b15d11209e45473e863766e5cd6e822f P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:55.103842 22984 tablet_copy_client.cc:647] T b15d11209e45473e863766e5cd6e822f P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:55.112367 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:55.153262 22984 tablet_copy_client.cc:523] T b15d11209e45473e863766e5cd6e822f P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:55.155908 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:26:55.180246 19117 tablet_copy_service.cc:140] P 007b2e79dc424a6f80141180f072f2d8: Received BeginTabletCopySession request for tablet 8937c809610542d88fdde3daf002c890 from peer 7ab6d395b30d4afb9821b61c9454430b ({username='slave'} at 127.5.107.71:48143)
I20240502 14:26:55.180478 19117 tablet_copy_service.cc:161] P 007b2e79dc424a6f80141180f072f2d8: Beginning new tablet copy session on tablet 8937c809610542d88fdde3daf002c890 from peer 7ab6d395b30d4afb9821b61c9454430b at {username='slave'} at 127.5.107.71:48143: session id = 7ab6d395b30d4afb9821b61c9454430b-8937c809610542d88fdde3daf002c890
I20240502 14:26:55.183265 19117 tablet_copy_source_session.cc:205] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:55.184566 21007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:55.184703 22988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8937c809610542d88fdde3daf002c890. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:55.192962 22988 tablet_copy_client.cc:783] T 8937c809610542d88fdde3daf002c890 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:55.193202 22988 tablet_copy_client.cc:647] T 8937c809610542d88fdde3daf002c890 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:55.200058 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:55.203960 22988 tablet_copy_client.cc:523] T 8937c809610542d88fdde3daf002c890 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:55.208567 22988 tablet_bootstrap.cc:492] T 8937c809610542d88fdde3daf002c890 P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap starting.
I20240502 14:26:55.189873 22984 tablet_bootstrap.cc:492] T b15d11209e45473e863766e5cd6e822f P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap starting.
I20240502 14:26:55.264695 22886 ts_tablet_manager.cc:902] T b13bcd6e693f4c7687a1aabdafbb83c6 P a834edffbcb2469aae598b069aab0681: Initiating tablet copy from peer c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:26:55.313014 22886 tablet_copy_client.cc:308] T b13bcd6e693f4c7687a1aabdafbb83c6 P a834edffbcb2469aae598b069aab0681: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.67:45409
I20240502 14:26:55.368101 20627 raft_consensus.cc:1203] T 097eb705c9e24740a5be61bf3354f3d8 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:26:55.389564 18985 tablet_copy_service.cc:140] P c902805060df4df9b5599d0e94b2dc06: Received BeginTabletCopySession request for tablet b13bcd6e693f4c7687a1aabdafbb83c6 from peer a834edffbcb2469aae598b069aab0681 ({username='slave'} at 127.5.107.69:33739)
I20240502 14:26:55.389829 18985 tablet_copy_service.cc:161] P c902805060df4df9b5599d0e94b2dc06: Beginning new tablet copy session on tablet b13bcd6e693f4c7687a1aabdafbb83c6 from peer a834edffbcb2469aae598b069aab0681 at {username='slave'} at 127.5.107.69:33739: session id = a834edffbcb2469aae598b069aab0681-b13bcd6e693f4c7687a1aabdafbb83c6
I20240502 14:26:55.390293 22988 tablet_bootstrap.cc:492] T 8937c809610542d88fdde3daf002c890 P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:55.390822 22988 tablet_bootstrap.cc:492] T 8937c809610542d88fdde3daf002c890 P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap complete.
I20240502 14:26:55.391150 22988 ts_tablet_manager.cc:1367] T 8937c809610542d88fdde3daf002c890 P 7ab6d395b30d4afb9821b61c9454430b: Time spent bootstrapping tablet: real 0.183s user 0.054s sys 0.006s
I20240502 14:26:55.392154 22988 raft_consensus.cc:348] T 8937c809610542d88fdde3daf002c890 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:55.392643 22988 raft_consensus.cc:729] T 8937c809610542d88fdde3daf002c890 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab6d395b30d4afb9821b61c9454430b, State: Initialized, Role: FOLLOWER
I20240502 14:26:55.392978 22988 consensus_queue.cc:260] T 8937c809610542d88fdde3daf002c890 P 7ab6d395b30d4afb9821b61c9454430b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:55.393682 22988 ts_tablet_manager.cc:1398] T 8937c809610542d88fdde3daf002c890 P 7ab6d395b30d4afb9821b61c9454430b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:26:55.394407 19117 tablet_copy_service.cc:342] P 007b2e79dc424a6f80141180f072f2d8: Request end of tablet copy session 7ab6d395b30d4afb9821b61c9454430b-8937c809610542d88fdde3daf002c890 received from {username='slave'} at 127.5.107.71:48143
I20240502 14:26:55.394527 19117 tablet_copy_service.cc:433] P 007b2e79dc424a6f80141180f072f2d8: ending tablet copy session 7ab6d395b30d4afb9821b61c9454430b-8937c809610542d88fdde3daf002c890 on tablet 8937c809610542d88fdde3daf002c890 with peer 7ab6d395b30d4afb9821b61c9454430b
I20240502 14:26:55.395140 18985 tablet_copy_source_session.cc:205] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:55.396749 22886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b13bcd6e693f4c7687a1aabdafbb83c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:55.403503 22886 tablet_copy_client.cc:783] T b13bcd6e693f4c7687a1aabdafbb83c6 P a834edffbcb2469aae598b069aab0681: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:55.403721 22886 tablet_copy_client.cc:647] T b13bcd6e693f4c7687a1aabdafbb83c6 P a834edffbcb2469aae598b069aab0681: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:55.405476 22886 tablet_copy_client.cc:523] T b13bcd6e693f4c7687a1aabdafbb83c6 P a834edffbcb2469aae598b069aab0681: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:55.409868 22886 tablet_bootstrap.cc:492] T b13bcd6e693f4c7687a1aabdafbb83c6 P a834edffbcb2469aae598b069aab0681: Bootstrap starting.
I20240502 14:26:55.416395 20627 raft_consensus.cc:1203] T 2c2afaffde96462eacd0d32bbdec23b1 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:26:55.450254 22984 tablet_bootstrap.cc:492] T b15d11209e45473e863766e5cd6e822f P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:55.453577 22984 tablet_bootstrap.cc:492] T b15d11209e45473e863766e5cd6e822f P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap complete.
I20240502 14:26:55.453892 22984 ts_tablet_manager.cc:1367] T b15d11209e45473e863766e5cd6e822f P ff6acba046404ef0a6f41ad5e09d852f: Time spent bootstrapping tablet: real 0.264s user 0.051s sys 0.008s
I20240502 14:26:55.456075 22984 raft_consensus.cc:348] T b15d11209e45473e863766e5cd6e822f P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:26:55.456323 22984 raft_consensus.cc:729] T b15d11209e45473e863766e5cd6e822f P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6acba046404ef0a6f41ad5e09d852f, State: Initialized, Role: FOLLOWER
I20240502 14:26:55.457480 22984 consensus_queue.cc:260] T b15d11209e45473e863766e5cd6e822f P ff6acba046404ef0a6f41ad5e09d852f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:26:55.476994 23047 ts_tablet_manager.cc:902] T 96662a3d9e1b4f03b05464ddd06b93d6 P a14c220ad2cc4bf9bd7dad65f4803eb0: Initiating tablet copy from peer 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:26:55.501091 22984 ts_tablet_manager.cc:1398] T b15d11209e45473e863766e5cd6e822f P ff6acba046404ef0a6f41ad5e09d852f: Time spent starting tablet: real 0.047s user 0.003s sys 0.001s
I20240502 14:26:55.506382 19117 tablet_copy_service.cc:342] P 007b2e79dc424a6f80141180f072f2d8: Request end of tablet copy session ff6acba046404ef0a6f41ad5e09d852f-b15d11209e45473e863766e5cd6e822f received from {username='slave'} at 127.5.107.72:33829
I20240502 14:26:55.506518 19117 tablet_copy_service.cc:433] P 007b2e79dc424a6f80141180f072f2d8: ending tablet copy session ff6acba046404ef0a6f41ad5e09d852f-b15d11209e45473e863766e5cd6e822f on tablet b15d11209e45473e863766e5cd6e822f with peer ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:26:55.541388 21029 raft_consensus.cc:1203] T b15d11209e45473e863766e5cd6e822f P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:26:55.553493 18833 consensus_queue.cc:237] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:26:55.563319 18965 raft_consensus.cc:1261] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:55.564011 22490 consensus_queue.cc:1035] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:55.571200 19093 raft_consensus.cc:1261] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:55.572582 22776 consensus_queue.cc:1035] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:26:55.573951 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5be0d6f4f1874c79b906c4bdc34b0ce9. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:55.583374 22490 raft_consensus.cc:2932] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:55.587054 18964 raft_consensus.cc:2932] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:55.600524 18554 catalog_manager.cc:5484] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06 reported cstate change: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:55.660406 19093 raft_consensus.cc:2932] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:55.708750 23047 tablet_copy_client.cc:308] T 96662a3d9e1b4f03b05464ddd06b93d6 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.65:36749
I20240502 14:26:55.709749 18721 tablet_copy_service.cc:140] P 2db304b538e14857b7b1cc272d0db94b: Received BeginTabletCopySession request for tablet 96662a3d9e1b4f03b05464ddd06b93d6 from peer a14c220ad2cc4bf9bd7dad65f4803eb0 ({username='slave'} at 127.5.107.70:48397)
I20240502 14:26:55.709946 18721 tablet_copy_service.cc:161] P 2db304b538e14857b7b1cc272d0db94b: Beginning new tablet copy session on tablet 96662a3d9e1b4f03b05464ddd06b93d6 from peer a14c220ad2cc4bf9bd7dad65f4803eb0 at {username='slave'} at 127.5.107.70:48397: session id = a14c220ad2cc4bf9bd7dad65f4803eb0-96662a3d9e1b4f03b05464ddd06b93d6
I20240502 14:26:55.713141 18721 tablet_copy_source_session.cc:205] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:55.721462 23047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96662a3d9e1b4f03b05464ddd06b93d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:55.734658 23047 tablet_copy_client.cc:783] T 96662a3d9e1b4f03b05464ddd06b93d6 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:55.735054 23047 tablet_copy_client.cc:647] T 96662a3d9e1b4f03b05464ddd06b93d6 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:55.746423 23047 tablet_copy_client.cc:523] T 96662a3d9e1b4f03b05464ddd06b93d6 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:55.755715 23047 tablet_bootstrap.cc:492] T 96662a3d9e1b4f03b05464ddd06b93d6 P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap starting.
I20240502 14:26:55.794224 22886 tablet_bootstrap.cc:492] T b13bcd6e693f4c7687a1aabdafbb83c6 P a834edffbcb2469aae598b069aab0681: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:55.796993 22886 tablet_bootstrap.cc:492] T b13bcd6e693f4c7687a1aabdafbb83c6 P a834edffbcb2469aae598b069aab0681: Bootstrap complete.
I20240502 14:26:55.798749 22886 ts_tablet_manager.cc:1367] T b13bcd6e693f4c7687a1aabdafbb83c6 P a834edffbcb2469aae598b069aab0681: Time spent bootstrapping tablet: real 0.389s user 0.068s sys 0.017s
I20240502 14:26:55.800711 22886 raft_consensus.cc:348] T b13bcd6e693f4c7687a1aabdafbb83c6 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:55.803576 22886 raft_consensus.cc:729] T b13bcd6e693f4c7687a1aabdafbb83c6 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a834edffbcb2469aae598b069aab0681, State: Initialized, Role: FOLLOWER
I20240502 14:26:55.804556 22886 consensus_queue.cc:260] T b13bcd6e693f4c7687a1aabdafbb83c6 P a834edffbcb2469aae598b069aab0681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:55.809537 22886 ts_tablet_manager.cc:1398] T b13bcd6e693f4c7687a1aabdafbb83c6 P a834edffbcb2469aae598b069aab0681: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240502 14:26:55.814750 18985 tablet_copy_service.cc:342] P c902805060df4df9b5599d0e94b2dc06: Request end of tablet copy session a834edffbcb2469aae598b069aab0681-b13bcd6e693f4c7687a1aabdafbb83c6 received from {username='slave'} at 127.5.107.69:33739
I20240502 14:26:55.814960 18985 tablet_copy_service.cc:433] P c902805060df4df9b5599d0e94b2dc06: ending tablet copy session a834edffbcb2469aae598b069aab0681-b13bcd6e693f4c7687a1aabdafbb83c6 on tablet b13bcd6e693f4c7687a1aabdafbb83c6 with peer a834edffbcb2469aae598b069aab0681
I20240502 14:26:55.945096 20894 raft_consensus.cc:1203] T 8937c809610542d88fdde3daf002c890 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 14:26:55.960860 23047 tablet_bootstrap.cc:492] T 96662a3d9e1b4f03b05464ddd06b93d6 P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:55.962271 23047 tablet_bootstrap.cc:492] T 96662a3d9e1b4f03b05464ddd06b93d6 P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap complete.
I20240502 14:26:55.962631 23047 ts_tablet_manager.cc:1367] T 96662a3d9e1b4f03b05464ddd06b93d6 P a14c220ad2cc4bf9bd7dad65f4803eb0: Time spent bootstrapping tablet: real 0.207s user 0.056s sys 0.012s
I20240502 14:26:55.963600 23047 raft_consensus.cc:348] T 96662a3d9e1b4f03b05464ddd06b93d6 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:26:55.964874 23047 raft_consensus.cc:729] T 96662a3d9e1b4f03b05464ddd06b93d6 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a14c220ad2cc4bf9bd7dad65f4803eb0, State: Initialized, Role: FOLLOWER
I20240502 14:26:55.965193 23047 consensus_queue.cc:260] T 96662a3d9e1b4f03b05464ddd06b93d6 P a14c220ad2cc4bf9bd7dad65f4803eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:26:55.968127 19097 consensus_queue.cc:237] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
W20240502 14:26:55.987766 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a992d12aee7416096d1353533397386. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:55.990725 18965 raft_consensus.cc:1261] T 6a992d12aee7416096d1353533397386 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:55.991159 18701 raft_consensus.cc:1261] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:55.991791 22792 consensus_queue.cc:1035] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:55.992553 22792 consensus_queue.cc:1035] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:56.015123 22786 raft_consensus.cc:2932] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:56.022486 18965 raft_consensus.cc:2932] T 6a992d12aee7416096d1353533397386 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:56.040874 18699 raft_consensus.cc:2932] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:56.042109 18556 catalog_manager.cc:5484] T 6a992d12aee7416096d1353533397386 P c902805060df4df9b5599d0e94b2dc06 reported cstate change: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:56.128670 18833 consensus_queue.cc:237] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:26:56.138370 20627 raft_consensus.cc:1261] T 2c2afaffde96462eacd0d32bbdec23b1 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:56.139225 23085 consensus_queue.cc:1035] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:56.146621 18965 raft_consensus.cc:1261] T 2c2afaffde96462eacd0d32bbdec23b1 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:56.147855 18699 raft_consensus.cc:1261] T 2c2afaffde96462eacd0d32bbdec23b1 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:56.148461 23085 consensus_queue.cc:1035] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:56.148989 22490 consensus_queue.cc:1035] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:56.153723 23149 ts_tablet_manager.cc:902] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P a14c220ad2cc4bf9bd7dad65f4803eb0: Initiating tablet copy from peer fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
I20240502 14:26:56.173638 19076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:26:56.195920 22490 raft_consensus.cc:2932] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:56.201087 18963 raft_consensus.cc:2932] T 2c2afaffde96462eacd0d32bbdec23b1 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:56.203848 18551 catalog_manager.cc:5484] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index 4 to 5, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:56.206131 23047 ts_tablet_manager.cc:1398] T 96662a3d9e1b4f03b05464ddd06b93d6 P a14c220ad2cc4bf9bd7dad65f4803eb0: Time spent starting tablet: real 0.240s user 0.003s sys 0.025s
I20240502 14:26:56.207883 18721 tablet_copy_service.cc:342] P 2db304b538e14857b7b1cc272d0db94b: Request end of tablet copy session a14c220ad2cc4bf9bd7dad65f4803eb0-96662a3d9e1b4f03b05464ddd06b93d6 received from {username='slave'} at 127.5.107.70:48397
I20240502 14:26:56.208032 18721 tablet_copy_service.cc:433] P 2db304b538e14857b7b1cc272d0db94b: ending tablet copy session a14c220ad2cc4bf9bd7dad65f4803eb0-96662a3d9e1b4f03b05464ddd06b93d6 on tablet 96662a3d9e1b4f03b05464ddd06b93d6 with peer a14c220ad2cc4bf9bd7dad65f4803eb0
W20240502 14:26:56.214499 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c2afaffde96462eacd0d32bbdec23b1. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:56.218093 18833 consensus_queue.cc:237] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:26:56.211489 18701 raft_consensus.cc:2932] T 2c2afaffde96462eacd0d32bbdec23b1 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:56.228318 20626 raft_consensus.cc:2932] T 2c2afaffde96462eacd0d32bbdec23b1 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:56.235390 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:56.241921 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:56.247728 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:26:56.252558 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67827c0262724897a05bf80df579bbf3. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:56.259701 18965 raft_consensus.cc:1261] T 67827c0262724897a05bf80df579bbf3 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:56.260581 22776 consensus_queue.cc:1035] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:56.269935 18701 raft_consensus.cc:1261] T 67827c0262724897a05bf80df579bbf3 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:56.270654 22776 consensus_queue.cc:1035] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:56.272831 19076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:26:56.309001 20627 raft_consensus.cc:1203] T b13bcd6e693f4c7687a1aabdafbb83c6 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:26:56.319583 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:56.321802 23149 tablet_copy_client.cc:308] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:33009
I20240502 14:26:56.323807 18854 tablet_copy_service.cc:140] P fec04c50dc5b49878ffa3c3873df32e4: Received BeginTabletCopySession request for tablet 5be0d6f4f1874c79b906c4bdc34b0ce9 from peer a14c220ad2cc4bf9bd7dad65f4803eb0 ({username='slave'} at 127.5.107.70:57723)
I20240502 14:26:56.323998 18854 tablet_copy_service.cc:161] P fec04c50dc5b49878ffa3c3873df32e4: Beginning new tablet copy session on tablet 5be0d6f4f1874c79b906c4bdc34b0ce9 from peer a14c220ad2cc4bf9bd7dad65f4803eb0 at {username='slave'} at 127.5.107.70:57723: session id = a14c220ad2cc4bf9bd7dad65f4803eb0-5be0d6f4f1874c79b906c4bdc34b0ce9
I20240502 14:26:56.327384 18854 tablet_copy_source_session.cc:205] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:56.336850 23149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be0d6f4f1874c79b906c4bdc34b0ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:56.369132 22776 raft_consensus.cc:2932] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:56.371785 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:56.375327 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:26:56.378813 18699 raft_consensus.cc:2932] T 67827c0262724897a05bf80df579bbf3 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:56.380071 18551 catalog_manager.cc:5484] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:56.381489 18965 raft_consensus.cc:2932] T 67827c0262724897a05bf80df579bbf3 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:56.389106 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:26:56.398936 21007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:56.464001 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:56.469486 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:56.471433 23149 tablet_copy_client.cc:783] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:56.471772 23149 tablet_copy_client.cc:647] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:56.484814 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:56.490556 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:56.506006 23149 tablet_copy_client.cc:523] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:56.510912 20760 raft_consensus.cc:1203] T 96662a3d9e1b4f03b05464ddd06b93d6 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:26:56.548483 23047 ts_tablet_manager.cc:902] T 6a992d12aee7416096d1353533397386 P a14c220ad2cc4bf9bd7dad65f4803eb0: Initiating tablet copy from peer 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:26:56.550331 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:26:56.569882 23149 tablet_bootstrap.cc:492] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap starting.
I20240502 14:26:56.573319 18699 consensus_queue.cc:237] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:56.599308 18832 raft_consensus.cc:1261] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:56.599351 19097 raft_consensus.cc:1261] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:26:56.600064 18637 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db56136f2dbb429098d4d78b15a3a439. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:56.600287 22983 consensus_queue.cc:1035] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:56.600633 21791 consensus_queue.cc:1035] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:56.614224 21007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:56.629158 22983 raft_consensus.cc:2932] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:56.645810 18551 catalog_manager.cc:5484] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b reported cstate change: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:56.645825 18832 raft_consensus.cc:2932] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:56.640872 19097 raft_consensus.cc:2932] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:56.671190 18832 consensus_queue.cc:237] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:56.675231 18699 raft_consensus.cc:1261] T 67827c0262724897a05bf80df579bbf3 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:26:56.676052 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67827c0262724897a05bf80df579bbf3. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:56.676486 22776 consensus_queue.cc:1035] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:26:56.676820 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67827c0262724897a05bf80df579bbf3. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:56.677839 18963 raft_consensus.cc:1261] T 67827c0262724897a05bf80df579bbf3 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:56.678467 22776 consensus_queue.cc:1035] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:56.681089 23047 tablet_copy_client.cc:308] T 6a992d12aee7416096d1353533397386 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.68:41463
I20240502 14:26:56.682015 19117 tablet_copy_service.cc:140] P 007b2e79dc424a6f80141180f072f2d8: Received BeginTabletCopySession request for tablet 6a992d12aee7416096d1353533397386 from peer a14c220ad2cc4bf9bd7dad65f4803eb0 ({username='slave'} at 127.5.107.70:47193)
I20240502 14:26:56.682221 19117 tablet_copy_service.cc:161] P 007b2e79dc424a6f80141180f072f2d8: Beginning new tablet copy session on tablet 6a992d12aee7416096d1353533397386 from peer a14c220ad2cc4bf9bd7dad65f4803eb0 at {username='slave'} at 127.5.107.70:47193: session id = a14c220ad2cc4bf9bd7dad65f4803eb0-6a992d12aee7416096d1353533397386
I20240502 14:26:56.686375 19117 tablet_copy_source_session.cc:205] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:56.687996 23047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a992d12aee7416096d1353533397386. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:56.692348 23085 raft_consensus.cc:2932] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:56.693269 18699 raft_consensus.cc:2932] T 67827c0262724897a05bf80df579bbf3 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:56.698485 18963 raft_consensus.cc:2932] T 67827c0262724897a05bf80df579bbf3 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:56.699873 18554 catalog_manager.cc:5484] T 67827c0262724897a05bf80df579bbf3 P 2db304b538e14857b7b1cc272d0db94b reported cstate change: config changed from index 4 to 5, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:56.724263 23047 tablet_copy_client.cc:783] T 6a992d12aee7416096d1353533397386 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:56.724560 23047 tablet_copy_client.cc:647] T 6a992d12aee7416096d1353533397386 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:56.727398 23047 tablet_copy_client.cc:523] T 6a992d12aee7416096d1353533397386 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:56.732894 23047 tablet_bootstrap.cc:492] T 6a992d12aee7416096d1353533397386 P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap starting.
I20240502 14:26:56.809720 23269 ts_tablet_manager.cc:902] T 2c2afaffde96462eacd0d32bbdec23b1 P ff6acba046404ef0a6f41ad5e09d852f: Initiating tablet copy from peer fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
I20240502 14:26:56.851730 23269 tablet_copy_client.cc:308] T 2c2afaffde96462eacd0d32bbdec23b1 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:33009
I20240502 14:26:56.852738 18854 tablet_copy_service.cc:140] P fec04c50dc5b49878ffa3c3873df32e4: Received BeginTabletCopySession request for tablet 2c2afaffde96462eacd0d32bbdec23b1 from peer ff6acba046404ef0a6f41ad5e09d852f ({username='slave'} at 127.5.107.72:58697)
I20240502 14:26:56.852933 18854 tablet_copy_service.cc:161] P fec04c50dc5b49878ffa3c3873df32e4: Beginning new tablet copy session on tablet 2c2afaffde96462eacd0d32bbdec23b1 from peer ff6acba046404ef0a6f41ad5e09d852f at {username='slave'} at 127.5.107.72:58697: session id = ff6acba046404ef0a6f41ad5e09d852f-2c2afaffde96462eacd0d32bbdec23b1
I20240502 14:26:56.857484 18854 tablet_copy_source_session.cc:205] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:56.862608 23269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c2afaffde96462eacd0d32bbdec23b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:56.881858 23269 tablet_copy_client.cc:783] T 2c2afaffde96462eacd0d32bbdec23b1 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:56.882330 23269 tablet_copy_client.cc:647] T 2c2afaffde96462eacd0d32bbdec23b1 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:56.886853 23269 tablet_copy_client.cc:523] T 2c2afaffde96462eacd0d32bbdec23b1 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:56.888417 23149 tablet_bootstrap.cc:492] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:56.888952 23149 tablet_bootstrap.cc:492] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap complete.
I20240502 14:26:56.889295 23149 ts_tablet_manager.cc:1367] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P a14c220ad2cc4bf9bd7dad65f4803eb0: Time spent bootstrapping tablet: real 0.320s user 0.059s sys 0.000s
I20240502 14:26:56.896440 23269 tablet_bootstrap.cc:492] T 2c2afaffde96462eacd0d32bbdec23b1 P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap starting.
I20240502 14:26:56.896107 23149 raft_consensus.cc:348] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:26:56.905429 23149 raft_consensus.cc:729] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a14c220ad2cc4bf9bd7dad65f4803eb0, State: Initialized, Role: FOLLOWER
I20240502 14:26:56.905773 23149 consensus_queue.cc:260] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P a14c220ad2cc4bf9bd7dad65f4803eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:26:56.907893 23149 ts_tablet_manager.cc:1398] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P a14c220ad2cc4bf9bd7dad65f4803eb0: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240502 14:26:56.908703 18854 tablet_copy_service.cc:342] P fec04c50dc5b49878ffa3c3873df32e4: Request end of tablet copy session a14c220ad2cc4bf9bd7dad65f4803eb0-5be0d6f4f1874c79b906c4bdc34b0ce9 received from {username='slave'} at 127.5.107.70:57723
I20240502 14:26:56.908859 18854 tablet_copy_service.cc:433] P fec04c50dc5b49878ffa3c3873df32e4: ending tablet copy session a14c220ad2cc4bf9bd7dad65f4803eb0-5be0d6f4f1874c79b906c4bdc34b0ce9 on tablet 5be0d6f4f1874c79b906c4bdc34b0ce9 with peer a14c220ad2cc4bf9bd7dad65f4803eb0
I20240502 14:26:56.927940 19097 consensus_queue.cc:237] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:26:56.933415 23047 tablet_bootstrap.cc:492] T 6a992d12aee7416096d1353533397386 P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:56.934372 23047 tablet_bootstrap.cc:492] T 6a992d12aee7416096d1353533397386 P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap complete.
I20240502 14:26:56.934669 23047 ts_tablet_manager.cc:1367] T 6a992d12aee7416096d1353533397386 P a14c220ad2cc4bf9bd7dad65f4803eb0: Time spent bootstrapping tablet: real 0.202s user 0.063s sys 0.001s
I20240502 14:26:56.935560 23047 raft_consensus.cc:348] T 6a992d12aee7416096d1353533397386 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:26:56.935822 23047 raft_consensus.cc:729] T 6a992d12aee7416096d1353533397386 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a14c220ad2cc4bf9bd7dad65f4803eb0, State: Initialized, Role: FOLLOWER
I20240502 14:26:56.936110 23047 consensus_queue.cc:260] T 6a992d12aee7416096d1353533397386 P a14c220ad2cc4bf9bd7dad65f4803eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:26:56.937456 23047 ts_tablet_manager.cc:1398] T 6a992d12aee7416096d1353533397386 P a14c220ad2cc4bf9bd7dad65f4803eb0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:26:56.937304 18965 consensus_queue.cc:237] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:56.938323 18963 raft_consensus.cc:1261] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:56.939976 18699 raft_consensus.cc:1261] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:56.940352 22792 consensus_queue.cc:1035] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:56.944528 19117 tablet_copy_service.cc:342] P 007b2e79dc424a6f80141180f072f2d8: Request end of tablet copy session a14c220ad2cc4bf9bd7dad65f4803eb0-6a992d12aee7416096d1353533397386 received from {username='slave'} at 127.5.107.70:47193
I20240502 14:26:56.944708 19117 tablet_copy_service.cc:433] P 007b2e79dc424a6f80141180f072f2d8: ending tablet copy session a14c220ad2cc4bf9bd7dad65f4803eb0-6a992d12aee7416096d1353533397386 on tablet 6a992d12aee7416096d1353533397386 with peer a14c220ad2cc4bf9bd7dad65f4803eb0
I20240502 14:26:56.946892 20760 raft_consensus.cc:1203] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:26:56.947690 23167 consensus_queue.cc:1035] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:26:56.958665 19031 consensus_peers.cc:485] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ecc6539b85d40e49df7ffc5f93cb801. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:26:56.970696 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ecb8a9b04ba34f779e2227e77e6f764c. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:56.970299 23167 raft_consensus.cc:2932] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:56.972225 18699 raft_consensus.cc:1261] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:56.973002 23236 consensus_queue.cc:1035] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:56.976039 18963 raft_consensus.cc:2932] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:56.981156 18832 raft_consensus.cc:1261] T ecb8a9b04ba34f779e2227e77e6f764c P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:56.982128 22912 consensus_queue.cc:1035] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:56.984552 18554 catalog_manager.cc:5484] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: config changed from index -1 to 4, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:56.986114 18700 raft_consensus.cc:2932] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:57.033478 23269 tablet_bootstrap.cc:492] T 2c2afaffde96462eacd0d32bbdec23b1 P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:57.033972 23269 tablet_bootstrap.cc:492] T 2c2afaffde96462eacd0d32bbdec23b1 P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap complete.
I20240502 14:26:57.034307 23269 ts_tablet_manager.cc:1367] T 2c2afaffde96462eacd0d32bbdec23b1 P ff6acba046404ef0a6f41ad5e09d852f: Time spent bootstrapping tablet: real 0.138s user 0.059s sys 0.002s
I20240502 14:26:57.035112 23269 raft_consensus.cc:348] T 2c2afaffde96462eacd0d32bbdec23b1 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:26:57.035411 23269 raft_consensus.cc:729] T 2c2afaffde96462eacd0d32bbdec23b1 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6acba046404ef0a6f41ad5e09d852f, State: Initialized, Role: FOLLOWER
I20240502 14:26:57.035701 23269 consensus_queue.cc:260] T 2c2afaffde96462eacd0d32bbdec23b1 P ff6acba046404ef0a6f41ad5e09d852f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
W20240502 14:26:57.038684 18898 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.65:36749, user_credentials={real_user=slave}} blocked reactor thread for 51059.6us
I20240502 14:26:57.039947 23320 raft_consensus.cc:2932] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:57.041110 18700 raft_consensus.cc:2932] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:57.042837 23269 ts_tablet_manager.cc:1398] T 2c2afaffde96462eacd0d32bbdec23b1 P ff6acba046404ef0a6f41ad5e09d852f: Time spent starting tablet: real 0.008s user 0.005s sys 0.003s
I20240502 14:26:57.043777 18854 tablet_copy_service.cc:342] P fec04c50dc5b49878ffa3c3873df32e4: Request end of tablet copy session ff6acba046404ef0a6f41ad5e09d852f-2c2afaffde96462eacd0d32bbdec23b1 received from {username='slave'} at 127.5.107.72:58697
I20240502 14:26:57.043928 18854 tablet_copy_service.cc:433] P fec04c50dc5b49878ffa3c3873df32e4: ending tablet copy session ff6acba046404ef0a6f41ad5e09d852f-2c2afaffde96462eacd0d32bbdec23b1 on tablet 2c2afaffde96462eacd0d32bbdec23b1 with peer ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:26:57.048012 18551 catalog_manager.cc:5484] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b reported cstate change: config changed from index -1 to 4, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:57.068316 23329 ts_tablet_manager.cc:902] T db56136f2dbb429098d4d78b15a3a439 P a834edffbcb2469aae598b069aab0681: Initiating tablet copy from peer 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:26:57.079298 18832 raft_consensus.cc:2932] T ecb8a9b04ba34f779e2227e77e6f764c P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:57.088943 20760 raft_consensus.cc:1203] T 6a992d12aee7416096d1353533397386 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:26:57.108230 23333 ts_tablet_manager.cc:902] T 67827c0262724897a05bf80df579bbf3 P 7ab6d395b30d4afb9821b61c9454430b: Initiating tablet copy from peer fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
I20240502 14:26:57.117331 23329 tablet_copy_client.cc:308] T db56136f2dbb429098d4d78b15a3a439 P a834edffbcb2469aae598b069aab0681: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.65:36749
I20240502 14:26:57.118345 18721 tablet_copy_service.cc:140] P 2db304b538e14857b7b1cc272d0db94b: Received BeginTabletCopySession request for tablet db56136f2dbb429098d4d78b15a3a439 from peer a834edffbcb2469aae598b069aab0681 ({username='slave'} at 127.5.107.69:53895)
I20240502 14:26:57.118631 18721 tablet_copy_service.cc:161] P 2db304b538e14857b7b1cc272d0db94b: Beginning new tablet copy session on tablet db56136f2dbb429098d4d78b15a3a439 from peer a834edffbcb2469aae598b069aab0681 at {username='slave'} at 127.5.107.69:53895: session id = a834edffbcb2469aae598b069aab0681-db56136f2dbb429098d4d78b15a3a439
I20240502 14:26:57.126896 18721 tablet_copy_source_session.cc:205] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:57.129801 23329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db56136f2dbb429098d4d78b15a3a439. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:57.136443 23333 tablet_copy_client.cc:308] T 67827c0262724897a05bf80df579bbf3 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:33009
I20240502 14:26:57.137457 18854 tablet_copy_service.cc:140] P fec04c50dc5b49878ffa3c3873df32e4: Received BeginTabletCopySession request for tablet 67827c0262724897a05bf80df579bbf3 from peer 7ab6d395b30d4afb9821b61c9454430b ({username='slave'} at 127.5.107.71:44135)
I20240502 14:26:57.137677 18854 tablet_copy_service.cc:161] P fec04c50dc5b49878ffa3c3873df32e4: Beginning new tablet copy session on tablet 67827c0262724897a05bf80df579bbf3 from peer 7ab6d395b30d4afb9821b61c9454430b at {username='slave'} at 127.5.107.71:44135: session id = 7ab6d395b30d4afb9821b61c9454430b-67827c0262724897a05bf80df579bbf3
I20240502 14:26:57.137771 23329 tablet_copy_client.cc:783] T db56136f2dbb429098d4d78b15a3a439 P a834edffbcb2469aae598b069aab0681: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:57.138010 23329 tablet_copy_client.cc:647] T db56136f2dbb429098d4d78b15a3a439 P a834edffbcb2469aae598b069aab0681: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:57.142575 18854 tablet_copy_source_session.cc:205] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:57.146891 23329 tablet_copy_client.cc:523] T db56136f2dbb429098d4d78b15a3a439 P a834edffbcb2469aae598b069aab0681: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:57.147696 23333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67827c0262724897a05bf80df579bbf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:57.150807 23329 tablet_bootstrap.cc:492] T db56136f2dbb429098d4d78b15a3a439 P a834edffbcb2469aae598b069aab0681: Bootstrap starting.
I20240502 14:26:57.151443 21028 raft_consensus.cc:1203] T 2c2afaffde96462eacd0d32bbdec23b1 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:26:57.167937 23333 tablet_copy_client.cc:783] T 67827c0262724897a05bf80df579bbf3 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:57.168284 23333 tablet_copy_client.cc:647] T 67827c0262724897a05bf80df579bbf3 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:57.171075 23333 tablet_copy_client.cc:523] T 67827c0262724897a05bf80df579bbf3 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:57.180713 23333 tablet_bootstrap.cc:492] T 67827c0262724897a05bf80df579bbf3 P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap starting.
I20240502 14:26:57.231624 23329 tablet_bootstrap.cc:492] T db56136f2dbb429098d4d78b15a3a439 P a834edffbcb2469aae598b069aab0681: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:57.232317 23329 tablet_bootstrap.cc:492] T db56136f2dbb429098d4d78b15a3a439 P a834edffbcb2469aae598b069aab0681: Bootstrap complete.
I20240502 14:26:57.232695 23329 ts_tablet_manager.cc:1367] T db56136f2dbb429098d4d78b15a3a439 P a834edffbcb2469aae598b069aab0681: Time spent bootstrapping tablet: real 0.082s user 0.053s sys 0.003s
I20240502 14:26:57.233668 23329 raft_consensus.cc:348] T db56136f2dbb429098d4d78b15a3a439 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:57.234031 23329 raft_consensus.cc:729] T db56136f2dbb429098d4d78b15a3a439 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a834edffbcb2469aae598b069aab0681, State: Initialized, Role: FOLLOWER
I20240502 14:26:57.234364 23329 consensus_queue.cc:260] T db56136f2dbb429098d4d78b15a3a439 P a834edffbcb2469aae598b069aab0681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:57.241597 23329 ts_tablet_manager.cc:1398] T db56136f2dbb429098d4d78b15a3a439 P a834edffbcb2469aae598b069aab0681: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20240502 14:26:57.244140 18721 tablet_copy_service.cc:342] P 2db304b538e14857b7b1cc272d0db94b: Request end of tablet copy session a834edffbcb2469aae598b069aab0681-db56136f2dbb429098d4d78b15a3a439 received from {username='slave'} at 127.5.107.69:53895
I20240502 14:26:57.244304 18721 tablet_copy_service.cc:433] P 2db304b538e14857b7b1cc272d0db94b: ending tablet copy session a834edffbcb2469aae598b069aab0681-db56136f2dbb429098d4d78b15a3a439 on tablet db56136f2dbb429098d4d78b15a3a439 with peer a834edffbcb2469aae598b069aab0681
I20240502 14:26:57.251873 23047 ts_tablet_manager.cc:902] T 67827c0262724897a05bf80df579bbf3 P a14c220ad2cc4bf9bd7dad65f4803eb0: Initiating tablet copy from peer fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
I20240502 14:26:57.312896 23047 tablet_copy_client.cc:308] T 67827c0262724897a05bf80df579bbf3 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:33009
I20240502 14:26:57.313886 18854 tablet_copy_service.cc:140] P fec04c50dc5b49878ffa3c3873df32e4: Received BeginTabletCopySession request for tablet 67827c0262724897a05bf80df579bbf3 from peer a14c220ad2cc4bf9bd7dad65f4803eb0 ({username='slave'} at 127.5.107.70:57723)
I20240502 14:26:57.314111 18854 tablet_copy_service.cc:161] P fec04c50dc5b49878ffa3c3873df32e4: Beginning new tablet copy session on tablet 67827c0262724897a05bf80df579bbf3 from peer a14c220ad2cc4bf9bd7dad65f4803eb0 at {username='slave'} at 127.5.107.70:57723: session id = a14c220ad2cc4bf9bd7dad65f4803eb0-67827c0262724897a05bf80df579bbf3
I20240502 14:26:57.329599 23269 ts_tablet_manager.cc:902] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f: Initiating tablet copy from peer 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:26:57.334962 18854 tablet_copy_source_session.cc:205] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:57.336942 23047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67827c0262724897a05bf80df579bbf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:57.354593 23047 tablet_copy_client.cc:783] T 67827c0262724897a05bf80df579bbf3 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:57.354938 23047 tablet_copy_client.cc:647] T 67827c0262724897a05bf80df579bbf3 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:57.357611 23047 tablet_copy_client.cc:523] T 67827c0262724897a05bf80df579bbf3 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:57.363871 23269 tablet_copy_client.cc:308] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.68:41463
I20240502 14:26:57.364794 19117 tablet_copy_service.cc:140] P 007b2e79dc424a6f80141180f072f2d8: Received BeginTabletCopySession request for tablet 1ecc6539b85d40e49df7ffc5f93cb801 from peer ff6acba046404ef0a6f41ad5e09d852f ({username='slave'} at 127.5.107.72:33829)
I20240502 14:26:57.365038 19117 tablet_copy_service.cc:161] P 007b2e79dc424a6f80141180f072f2d8: Beginning new tablet copy session on tablet 1ecc6539b85d40e49df7ffc5f93cb801 from peer ff6acba046404ef0a6f41ad5e09d852f at {username='slave'} at 127.5.107.72:33829: session id = ff6acba046404ef0a6f41ad5e09d852f-1ecc6539b85d40e49df7ffc5f93cb801
I20240502 14:26:57.368081 19117 tablet_copy_source_session.cc:205] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:57.369741 23269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ecc6539b85d40e49df7ffc5f93cb801. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:57.372469 23047 tablet_bootstrap.cc:492] T 67827c0262724897a05bf80df579bbf3 P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap starting.
I20240502 14:26:57.379971 23269 tablet_copy_client.cc:783] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:57.380223 23269 tablet_copy_client.cc:647] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:57.382064 23269 tablet_copy_client.cc:523] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:57.385900 19097 consensus_queue.cc:237] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:57.385699 23269 tablet_bootstrap.cc:492] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap starting.
W20240502 14:26:57.400532 19032 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a992d12aee7416096d1353533397386. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:57.404088 20760 raft_consensus.cc:1261] T 6a992d12aee7416096d1353533397386 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:57.404661 18700 raft_consensus.cc:1261] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:57.405110 23167 consensus_queue.cc:1035] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:57.406905 18963 raft_consensus.cc:1261] T 6a992d12aee7416096d1353533397386 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:57.407860 23311 consensus_queue.cc:1035] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:57.415042 22792 consensus_queue.cc:1035] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:57.441359 23420 ts_tablet_manager.cc:902] T ecb8a9b04ba34f779e2227e77e6f764c P 7ab6d395b30d4afb9821b61c9454430b: Initiating tablet copy from peer c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:26:57.465006 23333 tablet_bootstrap.cc:492] T 67827c0262724897a05bf80df579bbf3 P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:57.466219 23333 tablet_bootstrap.cc:492] T 67827c0262724897a05bf80df579bbf3 P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap complete.
I20240502 14:26:57.466650 23333 ts_tablet_manager.cc:1367] T 67827c0262724897a05bf80df579bbf3 P 7ab6d395b30d4afb9821b61c9454430b: Time spent bootstrapping tablet: real 0.286s user 0.070s sys 0.012s
I20240502 14:26:57.467808 23333 raft_consensus.cc:348] T 67827c0262724897a05bf80df579bbf3 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:57.468298 23333 raft_consensus.cc:729] T 67827c0262724897a05bf80df579bbf3 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab6d395b30d4afb9821b61c9454430b, State: Initialized, Role: FOLLOWER
I20240502 14:26:57.468680 23333 consensus_queue.cc:260] T 67827c0262724897a05bf80df579bbf3 P 7ab6d395b30d4afb9821b61c9454430b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:57.491452 23167 raft_consensus.cc:2932] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:57.495329 18699 raft_consensus.cc:2932] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:57.507226 18551 catalog_manager.cc:5484] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b reported cstate change: config changed from index 4 to 5, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:57.506669 20760 raft_consensus.cc:2932] T 6a992d12aee7416096d1353533397386 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:57.493288 23420 tablet_copy_client.cc:308] T ecb8a9b04ba34f779e2227e77e6f764c P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.67:45409
I20240502 14:26:57.514107 18985 tablet_copy_service.cc:140] P c902805060df4df9b5599d0e94b2dc06: Received BeginTabletCopySession request for tablet ecb8a9b04ba34f779e2227e77e6f764c from peer 7ab6d395b30d4afb9821b61c9454430b ({username='slave'} at 127.5.107.71:55147)
I20240502 14:26:57.514345 18985 tablet_copy_service.cc:161] P c902805060df4df9b5599d0e94b2dc06: Beginning new tablet copy session on tablet ecb8a9b04ba34f779e2227e77e6f764c from peer 7ab6d395b30d4afb9821b61c9454430b at {username='slave'} at 127.5.107.71:55147: session id = 7ab6d395b30d4afb9821b61c9454430b-ecb8a9b04ba34f779e2227e77e6f764c
I20240502 14:26:57.518467 18963 raft_consensus.cc:2932] T 6a992d12aee7416096d1353533397386 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:57.519012 18985 tablet_copy_source_session.cc:205] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:57.521101 23420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecb8a9b04ba34f779e2227e77e6f764c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:57.539271 23333 ts_tablet_manager.cc:1398] T 67827c0262724897a05bf80df579bbf3 P 7ab6d395b30d4afb9821b61c9454430b: Time spent starting tablet: real 0.072s user 0.002s sys 0.000s
I20240502 14:26:57.539749 23420 tablet_copy_client.cc:783] T ecb8a9b04ba34f779e2227e77e6f764c P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:57.540024 23420 tablet_copy_client.cc:647] T ecb8a9b04ba34f779e2227e77e6f764c P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:57.544075 18854 tablet_copy_service.cc:342] P fec04c50dc5b49878ffa3c3873df32e4: Request end of tablet copy session 7ab6d395b30d4afb9821b61c9454430b-67827c0262724897a05bf80df579bbf3 received from {username='slave'} at 127.5.107.71:44135
I20240502 14:26:57.544178 23047 tablet_bootstrap.cc:492] T 67827c0262724897a05bf80df579bbf3 P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:57.544250 18854 tablet_copy_service.cc:433] P fec04c50dc5b49878ffa3c3873df32e4: ending tablet copy session 7ab6d395b30d4afb9821b61c9454430b-67827c0262724897a05bf80df579bbf3 on tablet 67827c0262724897a05bf80df579bbf3 with peer 7ab6d395b30d4afb9821b61c9454430b
I20240502 14:26:57.545264 23420 tablet_copy_client.cc:523] T ecb8a9b04ba34f779e2227e77e6f764c P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:57.546427 23047 tablet_bootstrap.cc:492] T 67827c0262724897a05bf80df579bbf3 P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap complete.
I20240502 14:26:57.546764 23047 ts_tablet_manager.cc:1367] T 67827c0262724897a05bf80df579bbf3 P a14c220ad2cc4bf9bd7dad65f4803eb0: Time spent bootstrapping tablet: real 0.174s user 0.064s sys 0.002s
I20240502 14:26:57.552588 23420 tablet_bootstrap.cc:492] T ecb8a9b04ba34f779e2227e77e6f764c P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap starting.
I20240502 14:26:57.558085 23047 raft_consensus.cc:348] T 67827c0262724897a05bf80df579bbf3 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:57.560657 23047 raft_consensus.cc:729] T 67827c0262724897a05bf80df579bbf3 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a14c220ad2cc4bf9bd7dad65f4803eb0, State: Initialized, Role: FOLLOWER
I20240502 14:26:57.561038 23047 consensus_queue.cc:260] T 67827c0262724897a05bf80df579bbf3 P a14c220ad2cc4bf9bd7dad65f4803eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:57.561802 23047 ts_tablet_manager.cc:1398] T 67827c0262724897a05bf80df579bbf3 P a14c220ad2cc4bf9bd7dad65f4803eb0: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240502 14:26:57.562727 18854 tablet_copy_service.cc:342] P fec04c50dc5b49878ffa3c3873df32e4: Request end of tablet copy session a14c220ad2cc4bf9bd7dad65f4803eb0-67827c0262724897a05bf80df579bbf3 received from {username='slave'} at 127.5.107.70:57723
I20240502 14:26:57.562927 18854 tablet_copy_service.cc:433] P fec04c50dc5b49878ffa3c3873df32e4: ending tablet copy session a14c220ad2cc4bf9bd7dad65f4803eb0-67827c0262724897a05bf80df579bbf3 on tablet 67827c0262724897a05bf80df579bbf3 with peer a14c220ad2cc4bf9bd7dad65f4803eb0
I20240502 14:26:57.566053 18963 consensus_queue.cc:237] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
W20240502 14:26:57.586975 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 246a8425400f4890a5264c4ae8a627c5. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:57.587312 19097 raft_consensus.cc:1261] T 246a8425400f4890a5264c4ae8a627c5 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:57.587422 18700 raft_consensus.cc:1261] T 246a8425400f4890a5264c4ae8a627c5 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:57.587929 22912 consensus_queue.cc:1035] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:57.589814 22912 consensus_queue.cc:1035] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:57.613085 22912 raft_consensus.cc:2932] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:57.618777 19097 raft_consensus.cc:2932] T 246a8425400f4890a5264c4ae8a627c5 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:57.627959 23269 tablet_bootstrap.cc:492] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:57.635289 18699 raft_consensus.cc:2932] T 246a8425400f4890a5264c4ae8a627c5 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:57.646066 18556 catalog_manager.cc:5484] T 246a8425400f4890a5264c4ae8a627c5 P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: config changed from index -1 to 4, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:26:57.655179 23269 tablet_bootstrap.cc:492] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap complete.
I20240502 14:26:57.656965 23269 ts_tablet_manager.cc:1367] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f: Time spent bootstrapping tablet: real 0.271s user 0.060s sys 0.003s
I20240502 14:26:57.658943 23269 raft_consensus.cc:348] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:26:57.667848 23269 raft_consensus.cc:729] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6acba046404ef0a6f41ad5e09d852f, State: Initialized, Role: FOLLOWER
I20240502 14:26:57.668684 23269 consensus_queue.cc:260] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:26:57.680572 20627 raft_consensus.cc:1203] T db56136f2dbb429098d4d78b15a3a439 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240502 14:26:57.685904 18898 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.65:36749, user_credentials={real_user=slave}} blocked reactor thread for 50204.4us
I20240502 14:26:57.702185 20894 raft_consensus.cc:1203] T 67827c0262724897a05bf80df579bbf3 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:26:57.716454 23269 ts_tablet_manager.cc:1398] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f: Time spent starting tablet: real 0.058s user 0.003s sys 0.005s
I20240502 14:26:57.721729 19117 tablet_copy_service.cc:342] P 007b2e79dc424a6f80141180f072f2d8: Request end of tablet copy session ff6acba046404ef0a6f41ad5e09d852f-1ecc6539b85d40e49df7ffc5f93cb801 received from {username='slave'} at 127.5.107.72:33829
I20240502 14:26:57.721966 19117 tablet_copy_service.cc:433] P 007b2e79dc424a6f80141180f072f2d8: ending tablet copy session ff6acba046404ef0a6f41ad5e09d852f-1ecc6539b85d40e49df7ffc5f93cb801 on tablet 1ecc6539b85d40e49df7ffc5f93cb801 with peer ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:26:57.734669 20760 raft_consensus.cc:1203] T 67827c0262724897a05bf80df579bbf3 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:26:57.784083 18965 consensus_queue.cc:237] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:57.787490 18832 raft_consensus.cc:1261] T ecb8a9b04ba34f779e2227e77e6f764c P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:57.791164 18699 raft_consensus.cc:1261] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:57.792084 22912 consensus_queue.cc:1035] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:26:57.794257 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ecb8a9b04ba34f779e2227e77e6f764c. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:57.796175 23320 consensus_queue.cc:1035] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:26:57.800107 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:57.803170 23320 raft_consensus.cc:2932] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:57.808135 18832 raft_consensus.cc:2932] T ecb8a9b04ba34f779e2227e77e6f764c P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:57.842311 18554 catalog_manager.cc:5484] T ecb8a9b04ba34f779e2227e77e6f764c P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index 4 to 5, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:57.843605 23420 tablet_bootstrap.cc:492] T ecb8a9b04ba34f779e2227e77e6f764c P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:57.844084 23420 tablet_bootstrap.cc:492] T ecb8a9b04ba34f779e2227e77e6f764c P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap complete.
I20240502 14:26:57.844381 23420 ts_tablet_manager.cc:1367] T ecb8a9b04ba34f779e2227e77e6f764c P 7ab6d395b30d4afb9821b61c9454430b: Time spent bootstrapping tablet: real 0.292s user 0.071s sys 0.031s
I20240502 14:26:57.845307 23420 raft_consensus.cc:348] T ecb8a9b04ba34f779e2227e77e6f764c P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:57.848484 23420 raft_consensus.cc:729] T ecb8a9b04ba34f779e2227e77e6f764c P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab6d395b30d4afb9821b61c9454430b, State: Initialized, Role: FOLLOWER
I20240502 14:26:57.848994 23420 consensus_queue.cc:260] T ecb8a9b04ba34f779e2227e77e6f764c P 7ab6d395b30d4afb9821b61c9454430b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:57.850625 23420 ts_tablet_manager.cc:1398] T ecb8a9b04ba34f779e2227e77e6f764c P 7ab6d395b30d4afb9821b61c9454430b: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 14:26:57.851970 18985 tablet_copy_service.cc:342] P c902805060df4df9b5599d0e94b2dc06: Request end of tablet copy session 7ab6d395b30d4afb9821b61c9454430b-ecb8a9b04ba34f779e2227e77e6f764c received from {username='slave'} at 127.5.107.71:55147
I20240502 14:26:57.852111 18985 tablet_copy_service.cc:433] P c902805060df4df9b5599d0e94b2dc06: ending tablet copy session 7ab6d395b30d4afb9821b61c9454430b-ecb8a9b04ba34f779e2227e77e6f764c on tablet ecb8a9b04ba34f779e2227e77e6f764c with peer 7ab6d395b30d4afb9821b61c9454430b
I20240502 14:26:57.896730 18699 raft_consensus.cc:2932] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:57.901261 21028 raft_consensus.cc:1203] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:26:57.945755 23511 ts_tablet_manager.cc:902] T 6a992d12aee7416096d1353533397386 P a834edffbcb2469aae598b069aab0681: Initiating tablet copy from peer 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:26:57.996351 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:26:58.000718 23047 ts_tablet_manager.cc:902] T 246a8425400f4890a5264c4ae8a627c5 P a14c220ad2cc4bf9bd7dad65f4803eb0: Initiating tablet copy from peer c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:26:58.101775 23511 tablet_copy_client.cc:308] T 6a992d12aee7416096d1353533397386 P a834edffbcb2469aae598b069aab0681: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.68:41463
I20240502 14:26:58.102670 19117 tablet_copy_service.cc:140] P 007b2e79dc424a6f80141180f072f2d8: Received BeginTabletCopySession request for tablet 6a992d12aee7416096d1353533397386 from peer a834edffbcb2469aae598b069aab0681 ({username='slave'} at 127.5.107.69:37525)
I20240502 14:26:58.102857 19117 tablet_copy_service.cc:161] P 007b2e79dc424a6f80141180f072f2d8: Beginning new tablet copy session on tablet 6a992d12aee7416096d1353533397386 from peer a834edffbcb2469aae598b069aab0681 at {username='slave'} at 127.5.107.69:37525: session id = a834edffbcb2469aae598b069aab0681-6a992d12aee7416096d1353533397386
I20240502 14:26:58.106544 19117 tablet_copy_source_session.cc:205] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:58.114800 23511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a992d12aee7416096d1353533397386. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:58.125492 23511 tablet_copy_client.cc:783] T 6a992d12aee7416096d1353533397386 P a834edffbcb2469aae598b069aab0681: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:58.125926 23511 tablet_copy_client.cc:647] T 6a992d12aee7416096d1353533397386 P a834edffbcb2469aae598b069aab0681: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:58.128065 23511 tablet_copy_client.cc:523] T 6a992d12aee7416096d1353533397386 P a834edffbcb2469aae598b069aab0681: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:58.133510 23511 tablet_bootstrap.cc:492] T 6a992d12aee7416096d1353533397386 P a834edffbcb2469aae598b069aab0681: Bootstrap starting.
I20240502 14:26:58.154577 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:26:58.161279 21007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:58.180732 23047 tablet_copy_client.cc:308] T 246a8425400f4890a5264c4ae8a627c5 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.67:45409
I20240502 14:26:58.181828 18985 tablet_copy_service.cc:140] P c902805060df4df9b5599d0e94b2dc06: Received BeginTabletCopySession request for tablet 246a8425400f4890a5264c4ae8a627c5 from peer a14c220ad2cc4bf9bd7dad65f4803eb0 ({username='slave'} at 127.5.107.70:56629)
I20240502 14:26:58.182067 18985 tablet_copy_service.cc:161] P c902805060df4df9b5599d0e94b2dc06: Beginning new tablet copy session on tablet 246a8425400f4890a5264c4ae8a627c5 from peer a14c220ad2cc4bf9bd7dad65f4803eb0 at {username='slave'} at 127.5.107.70:56629: session id = a14c220ad2cc4bf9bd7dad65f4803eb0-246a8425400f4890a5264c4ae8a627c5
I20240502 14:26:58.185473 19076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:26:58.186872 18985 tablet_copy_source_session.cc:205] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:58.191910 23047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 246a8425400f4890a5264c4ae8a627c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:58.217944 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:58.220472 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:58.237908 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:58.243657 23559 ts_tablet_manager.cc:902] T ecb8a9b04ba34f779e2227e77e6f764c P a834edffbcb2469aae598b069aab0681: Initiating tablet copy from peer c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:26:58.249915 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:58.262048 19076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:26:58.290349 23047 tablet_copy_client.cc:783] T 246a8425400f4890a5264c4ae8a627c5 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:58.293442 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:26:58.319031 23047 tablet_copy_client.cc:647] T 246a8425400f4890a5264c4ae8a627c5 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:58.324326 23047 tablet_copy_client.cc:523] T 246a8425400f4890a5264c4ae8a627c5 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:58.331048 20894 raft_consensus.cc:2932] T ecb8a9b04ba34f779e2227e77e6f764c P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:58.346371 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:58.348884 23047 tablet_bootstrap.cc:492] T 246a8425400f4890a5264c4ae8a627c5 P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap starting.
I20240502 14:26:58.350706 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:58.381781 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:58.387465 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:26:58.420542 23559 tablet_copy_client.cc:308] T ecb8a9b04ba34f779e2227e77e6f764c P a834edffbcb2469aae598b069aab0681: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.67:45409
I20240502 14:26:58.423928 18985 tablet_copy_service.cc:140] P c902805060df4df9b5599d0e94b2dc06: Received BeginTabletCopySession request for tablet ecb8a9b04ba34f779e2227e77e6f764c from peer a834edffbcb2469aae598b069aab0681 ({username='slave'} at 127.5.107.69:33739)
I20240502 14:26:58.424110 18985 tablet_copy_service.cc:161] P c902805060df4df9b5599d0e94b2dc06: Beginning new tablet copy session on tablet ecb8a9b04ba34f779e2227e77e6f764c from peer a834edffbcb2469aae598b069aab0681 at {username='slave'} at 127.5.107.69:33739: session id = a834edffbcb2469aae598b069aab0681-ecb8a9b04ba34f779e2227e77e6f764c
I20240502 14:26:58.425335 21007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:58.425483 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:58.427299 18985 tablet_copy_source_session.cc:205] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:58.428773 23559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecb8a9b04ba34f779e2227e77e6f764c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:58.514307 23511 tablet_bootstrap.cc:492] T 6a992d12aee7416096d1353533397386 P a834edffbcb2469aae598b069aab0681: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:58.514868 23511 tablet_bootstrap.cc:492] T 6a992d12aee7416096d1353533397386 P a834edffbcb2469aae598b069aab0681: Bootstrap complete.
I20240502 14:26:58.515287 23511 ts_tablet_manager.cc:1367] T 6a992d12aee7416096d1353533397386 P a834edffbcb2469aae598b069aab0681: Time spent bootstrapping tablet: real 0.382s user 0.067s sys 0.013s
I20240502 14:26:58.516259 23559 tablet_copy_client.cc:783] T ecb8a9b04ba34f779e2227e77e6f764c P a834edffbcb2469aae598b069aab0681: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:58.516122 23511 raft_consensus.cc:348] T 6a992d12aee7416096d1353533397386 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:58.516512 23559 tablet_copy_client.cc:647] T ecb8a9b04ba34f779e2227e77e6f764c P a834edffbcb2469aae598b069aab0681: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:58.516537 23511 raft_consensus.cc:729] T 6a992d12aee7416096d1353533397386 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a834edffbcb2469aae598b069aab0681, State: Initialized, Role: FOLLOWER
I20240502 14:26:58.516839 23511 consensus_queue.cc:260] T 6a992d12aee7416096d1353533397386 P a834edffbcb2469aae598b069aab0681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:58.520967 23559 tablet_copy_client.cc:523] T ecb8a9b04ba34f779e2227e77e6f764c P a834edffbcb2469aae598b069aab0681: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:58.552357 23559 tablet_bootstrap.cc:492] T ecb8a9b04ba34f779e2227e77e6f764c P a834edffbcb2469aae598b069aab0681: Bootstrap starting.
I20240502 14:26:58.553777 23511 ts_tablet_manager.cc:1398] T 6a992d12aee7416096d1353533397386 P a834edffbcb2469aae598b069aab0681: Time spent starting tablet: real 0.038s user 0.007s sys 0.004s
I20240502 14:26:58.559278 19117 tablet_copy_service.cc:342] P 007b2e79dc424a6f80141180f072f2d8: Request end of tablet copy session a834edffbcb2469aae598b069aab0681-6a992d12aee7416096d1353533397386 received from {username='slave'} at 127.5.107.69:37525
I20240502 14:26:58.559468 19117 tablet_copy_service.cc:433] P 007b2e79dc424a6f80141180f072f2d8: ending tablet copy session a834edffbcb2469aae598b069aab0681-6a992d12aee7416096d1353533397386 on tablet 6a992d12aee7416096d1353533397386 with peer a834edffbcb2469aae598b069aab0681
I20240502 14:26:58.572925 18699 consensus_queue.cc:237] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:26:58.585717 20626 raft_consensus.cc:1261] T db56136f2dbb429098d4d78b15a3a439 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:58.586117 18833 raft_consensus.cc:1261] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:26:58.586737 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db56136f2dbb429098d4d78b15a3a439. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:58.586861 23493 consensus_queue.cc:1035] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:58.588658 21791 consensus_queue.cc:1035] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:58.613442 19097 raft_consensus.cc:1261] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:58.614073 21791 consensus_queue.cc:1035] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:58.704540 21791 raft_consensus.cc:2932] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:58.710531 19093 raft_consensus.cc:2932] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:58.726110 18551 catalog_manager.cc:5484] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: config changed from index 4 to 5, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:58.730446 23047 tablet_bootstrap.cc:492] T 246a8425400f4890a5264c4ae8a627c5 P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:58.730967 23047 tablet_bootstrap.cc:492] T 246a8425400f4890a5264c4ae8a627c5 P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap complete.
I20240502 14:26:58.731266 23047 ts_tablet_manager.cc:1367] T 246a8425400f4890a5264c4ae8a627c5 P a14c220ad2cc4bf9bd7dad65f4803eb0: Time spent bootstrapping tablet: real 0.382s user 0.039s sys 0.015s
I20240502 14:26:58.737347 18833 raft_consensus.cc:2932] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:58.760769 20626 raft_consensus.cc:2932] T db56136f2dbb429098d4d78b15a3a439 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:58.779012 23047 raft_consensus.cc:348] T 246a8425400f4890a5264c4ae8a627c5 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:26:58.779408 23047 raft_consensus.cc:729] T 246a8425400f4890a5264c4ae8a627c5 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a14c220ad2cc4bf9bd7dad65f4803eb0, State: Initialized, Role: FOLLOWER
I20240502 14:26:58.779732 23047 consensus_queue.cc:260] T 246a8425400f4890a5264c4ae8a627c5 P a14c220ad2cc4bf9bd7dad65f4803eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:26:58.785976 23047 ts_tablet_manager.cc:1398] T 246a8425400f4890a5264c4ae8a627c5 P a14c220ad2cc4bf9bd7dad65f4803eb0: Time spent starting tablet: real 0.055s user 0.004s sys 0.000s
I20240502 14:26:58.786862 18985 tablet_copy_service.cc:342] P c902805060df4df9b5599d0e94b2dc06: Request end of tablet copy session a14c220ad2cc4bf9bd7dad65f4803eb0-246a8425400f4890a5264c4ae8a627c5 received from {username='slave'} at 127.5.107.70:56629
I20240502 14:26:58.787010 18985 tablet_copy_service.cc:433] P c902805060df4df9b5599d0e94b2dc06: ending tablet copy session a14c220ad2cc4bf9bd7dad65f4803eb0-246a8425400f4890a5264c4ae8a627c5 on tablet 246a8425400f4890a5264c4ae8a627c5 with peer a14c220ad2cc4bf9bd7dad65f4803eb0
I20240502 14:26:58.836537 20626 raft_consensus.cc:1203] T 6a992d12aee7416096d1353533397386 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:26:58.964672 23559 tablet_bootstrap.cc:492] T ecb8a9b04ba34f779e2227e77e6f764c P a834edffbcb2469aae598b069aab0681: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:58.967607 23559 tablet_bootstrap.cc:492] T ecb8a9b04ba34f779e2227e77e6f764c P a834edffbcb2469aae598b069aab0681: Bootstrap complete.
I20240502 14:26:58.968869 23559 ts_tablet_manager.cc:1367] T ecb8a9b04ba34f779e2227e77e6f764c P a834edffbcb2469aae598b069aab0681: Time spent bootstrapping tablet: real 0.417s user 0.051s sys 0.012s
I20240502 14:26:58.975157 23559 raft_consensus.cc:348] T ecb8a9b04ba34f779e2227e77e6f764c P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:58.979192 23559 raft_consensus.cc:729] T ecb8a9b04ba34f779e2227e77e6f764c P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a834edffbcb2469aae598b069aab0681, State: Initialized, Role: FOLLOWER
I20240502 14:26:58.980418 23559 consensus_queue.cc:260] T ecb8a9b04ba34f779e2227e77e6f764c P a834edffbcb2469aae598b069aab0681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:58.988639 23559 ts_tablet_manager.cc:1398] T ecb8a9b04ba34f779e2227e77e6f764c P a834edffbcb2469aae598b069aab0681: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240502 14:26:58.989984 18985 tablet_copy_service.cc:342] P c902805060df4df9b5599d0e94b2dc06: Request end of tablet copy session a834edffbcb2469aae598b069aab0681-ecb8a9b04ba34f779e2227e77e6f764c received from {username='slave'} at 127.5.107.69:33739
I20240502 14:26:58.990120 18985 tablet_copy_service.cc:433] P c902805060df4df9b5599d0e94b2dc06: ending tablet copy session a834edffbcb2469aae598b069aab0681-ecb8a9b04ba34f779e2227e77e6f764c on tablet ecb8a9b04ba34f779e2227e77e6f764c with peer a834edffbcb2469aae598b069aab0681
I20240502 14:26:59.009382 23656 ts_tablet_manager.cc:902] T db56136f2dbb429098d4d78b15a3a439 P ff6acba046404ef0a6f41ad5e09d852f: Initiating tablet copy from peer 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:26:59.046394 20760 raft_consensus.cc:1203] T 246a8425400f4890a5264c4ae8a627c5 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:26:59.111413 18965 consensus_queue.cc:237] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:59.115281 23656 tablet_copy_client.cc:308] T db56136f2dbb429098d4d78b15a3a439 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.65:36749
I20240502 14:26:59.115548 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:59.116117 18721 tablet_copy_service.cc:140] P 2db304b538e14857b7b1cc272d0db94b: Received BeginTabletCopySession request for tablet db56136f2dbb429098d4d78b15a3a439 from peer ff6acba046404ef0a6f41ad5e09d852f ({username='slave'} at 127.5.107.72:37745)
I20240502 14:26:59.116292 18721 tablet_copy_service.cc:161] P 2db304b538e14857b7b1cc272d0db94b: Beginning new tablet copy session on tablet db56136f2dbb429098d4d78b15a3a439 from peer ff6acba046404ef0a6f41ad5e09d852f at {username='slave'} at 127.5.107.72:37745: session id = ff6acba046404ef0a6f41ad5e09d852f-db56136f2dbb429098d4d78b15a3a439
I20240502 14:26:59.119378 18721 tablet_copy_source_session.cc:205] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:59.121229 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:26:59.121853 23656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db56136f2dbb429098d4d78b15a3a439. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:59.126725 19097 raft_consensus.cc:1261] T 246a8425400f4890a5264c4ae8a627c5 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:59.127540 23616 consensus_queue.cc:1035] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:26:59.130385 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 246a8425400f4890a5264c4ae8a627c5. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:59.130429 20760 raft_consensus.cc:1261] T 246a8425400f4890a5264c4ae8a627c5 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:59.135278 23616 consensus_queue.cc:1035] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:59.182112 18699 raft_consensus.cc:1261] T 246a8425400f4890a5264c4ae8a627c5 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:59.182585 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:26:59.183655 23616 consensus_queue.cc:1035] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:59.192836 23616 raft_consensus.cc:2932] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:59.204479 18554 catalog_manager.cc:5484] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 reported cstate change: config changed from index 4 to 5, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:59.216223 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:59.222609 18699 raft_consensus.cc:2932] T 246a8425400f4890a5264c4ae8a627c5 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:59.226800 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:59.227558 20627 raft_consensus.cc:1203] T ecb8a9b04ba34f779e2227e77e6f764c P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:26:59.243310 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:59.242564 20760 raft_consensus.cc:2932] T 246a8425400f4890a5264c4ae8a627c5 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:26:59.260100 19076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:26:59.260982 19093 raft_consensus.cc:2932] T 246a8425400f4890a5264c4ae8a627c5 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
W20240502 14:26:59.275142 18769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.68:41463, user_credentials={real_user=slave}} blocked reactor thread for 44438.9us
I20240502 14:26:59.290925 21007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:59.347569 23656 tablet_copy_client.cc:783] T db56136f2dbb429098d4d78b15a3a439 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:59.350785 23656 tablet_copy_client.cc:647] T db56136f2dbb429098d4d78b15a3a439 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:59.350927 19076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:26:59.354570 23656 tablet_copy_client.cc:523] T db56136f2dbb429098d4d78b15a3a439 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:59.378876 23656 tablet_bootstrap.cc:492] T db56136f2dbb429098d4d78b15a3a439 P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap starting.
I20240502 14:26:59.397080 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:59.409972 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:59.418367 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:26:59.425990 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:59.481518 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:26:59.509827 21007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:26:59.602085 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:26:59.606069 18699 consensus_queue.cc:237] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:26:59.606860 23656 tablet_bootstrap.cc:492] T db56136f2dbb429098d4d78b15a3a439 P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:59.607386 23656 tablet_bootstrap.cc:492] T db56136f2dbb429098d4d78b15a3a439 P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap complete.
I20240502 14:26:59.607673 23656 ts_tablet_manager.cc:1367] T db56136f2dbb429098d4d78b15a3a439 P ff6acba046404ef0a6f41ad5e09d852f: Time spent bootstrapping tablet: real 0.229s user 0.050s sys 0.005s
I20240502 14:26:59.621145 23656 raft_consensus.cc:348] T db56136f2dbb429098d4d78b15a3a439 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:26:59.621502 23656 raft_consensus.cc:729] T db56136f2dbb429098d4d78b15a3a439 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6acba046404ef0a6f41ad5e09d852f, State: Initialized, Role: FOLLOWER
I20240502 14:26:59.621788 23656 consensus_queue.cc:260] T db56136f2dbb429098d4d78b15a3a439 P ff6acba046404ef0a6f41ad5e09d852f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:26:59.629272 23656 ts_tablet_manager.cc:1398] T db56136f2dbb429098d4d78b15a3a439 P ff6acba046404ef0a6f41ad5e09d852f: Time spent starting tablet: real 0.021s user 0.004s sys 0.006s
I20240502 14:26:59.631203 18721 tablet_copy_service.cc:342] P 2db304b538e14857b7b1cc272d0db94b: Request end of tablet copy session ff6acba046404ef0a6f41ad5e09d852f-db56136f2dbb429098d4d78b15a3a439 received from {username='slave'} at 127.5.107.72:37745
I20240502 14:26:59.631338 18721 tablet_copy_service.cc:433] P 2db304b538e14857b7b1cc272d0db94b: ending tablet copy session ff6acba046404ef0a6f41ad5e09d852f-db56136f2dbb429098d4d78b15a3a439 on tablet db56136f2dbb429098d4d78b15a3a439 with peer ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:26:59.676928 18965 raft_consensus.cc:1261] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:26:59.677026 18637 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 51ef0be93c914e258635e13f3cca79aa. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:59.677747 23493 consensus_queue.cc:1035] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:59.678381 18833 raft_consensus.cc:1261] T 51ef0be93c914e258635e13f3cca79aa P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:26:59.679069 23608 consensus_queue.cc:1035] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:26:59.694491 23493 raft_consensus.cc:2932] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:59.695399 18965 raft_consensus.cc:2932] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:59.702852 18553 catalog_manager.cc:5484] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06 reported cstate change: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:59.706769 18832 raft_consensus.cc:2932] T 51ef0be93c914e258635e13f3cca79aa P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:26:59.744508 23768 ts_tablet_manager.cc:902] T 246a8425400f4890a5264c4ae8a627c5 P 7ab6d395b30d4afb9821b61c9454430b: Initiating tablet copy from peer c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:26:59.827950 23768 tablet_copy_client.cc:308] T 246a8425400f4890a5264c4ae8a627c5 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.67:45409
I20240502 14:26:59.829241 18985 tablet_copy_service.cc:140] P c902805060df4df9b5599d0e94b2dc06: Received BeginTabletCopySession request for tablet 246a8425400f4890a5264c4ae8a627c5 from peer 7ab6d395b30d4afb9821b61c9454430b ({username='slave'} at 127.5.107.71:55147)
I20240502 14:26:59.829488 18985 tablet_copy_service.cc:161] P c902805060df4df9b5599d0e94b2dc06: Beginning new tablet copy session on tablet 246a8425400f4890a5264c4ae8a627c5 from peer 7ab6d395b30d4afb9821b61c9454430b at {username='slave'} at 127.5.107.71:55147: session id = 7ab6d395b30d4afb9821b61c9454430b-246a8425400f4890a5264c4ae8a627c5
I20240502 14:26:59.832695 18985 tablet_copy_source_session.cc:205] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:26:59.837826 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 246a8425400f4890a5264c4ae8a627c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:26:59.867240 23768 tablet_copy_client.cc:783] T 246a8425400f4890a5264c4ae8a627c5 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Starting download of 0 data blocks...
I20240502 14:26:59.868734 23768 tablet_copy_client.cc:647] T 246a8425400f4890a5264c4ae8a627c5 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:26:59.875903 23768 tablet_copy_client.cc:523] T 246a8425400f4890a5264c4ae8a627c5 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:26:59.885555 23768 tablet_bootstrap.cc:492] T 246a8425400f4890a5264c4ae8a627c5 P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap starting.
I20240502 14:26:59.966853 18699 consensus_queue.cc:237] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
W20240502 14:26:59.972769 18637 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 51ef0be93c914e258635e13f3cca79aa. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:59.972855 18965 raft_consensus.cc:1261] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:26:59.974478 18636 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 51ef0be93c914e258635e13f3cca79aa. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:26:59.975334 23608 consensus_queue.cc:1035] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:59.977988 18833 raft_consensus.cc:1261] T 51ef0be93c914e258635e13f3cca79aa P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:26:59.978715 23493 consensus_queue.cc:1035] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:26:59.981606 23608 raft_consensus.cc:2932] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:59.986512 23768 tablet_bootstrap.cc:492] T 246a8425400f4890a5264c4ae8a627c5 P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:26:59.986979 23768 tablet_bootstrap.cc:492] T 246a8425400f4890a5264c4ae8a627c5 P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap complete.
I20240502 14:26:59.986515 18832 raft_consensus.cc:2932] T 51ef0be93c914e258635e13f3cca79aa P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:59.987270 23768 ts_tablet_manager.cc:1367] T 246a8425400f4890a5264c4ae8a627c5 P 7ab6d395b30d4afb9821b61c9454430b: Time spent bootstrapping tablet: real 0.102s user 0.057s sys 0.008s
I20240502 14:26:59.988150 23768 raft_consensus.cc:348] T 246a8425400f4890a5264c4ae8a627c5 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:59.988410 23768 raft_consensus.cc:729] T 246a8425400f4890a5264c4ae8a627c5 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab6d395b30d4afb9821b61c9454430b, State: Initialized, Role: FOLLOWER
I20240502 14:26:59.989012 23768 consensus_queue.cc:260] T 246a8425400f4890a5264c4ae8a627c5 P 7ab6d395b30d4afb9821b61c9454430b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:59.994424 18553 catalog_manager.cc:5484] T 51ef0be93c914e258635e13f3cca79aa P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index 4 to 5, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:26:59.999567 23768 ts_tablet_manager.cc:1398] T 246a8425400f4890a5264c4ae8a627c5 P 7ab6d395b30d4afb9821b61c9454430b: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20240502 14:27:00.001646 18985 tablet_copy_service.cc:342] P c902805060df4df9b5599d0e94b2dc06: Request end of tablet copy session 7ab6d395b30d4afb9821b61c9454430b-246a8425400f4890a5264c4ae8a627c5 received from {username='slave'} at 127.5.107.71:55147
I20240502 14:27:00.001788 18985 tablet_copy_service.cc:433] P c902805060df4df9b5599d0e94b2dc06: ending tablet copy session 7ab6d395b30d4afb9821b61c9454430b-246a8425400f4890a5264c4ae8a627c5 on tablet 246a8425400f4890a5264c4ae8a627c5 with peer 7ab6d395b30d4afb9821b61c9454430b
I20240502 14:27:00.010314 18965 raft_consensus.cc:2932] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:00.039163 21029 raft_consensus.cc:1203] T db56136f2dbb429098d4d78b15a3a439 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:27:00.071023 18699 consensus_queue.cc:237] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:27:00.091836 18832 raft_consensus.cc:1261] T d28db16c9f7e405b8866f0e6c548e246 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:27:00.092422 19097 raft_consensus.cc:1261] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:27:00.093803 23828 consensus_queue.cc:1035] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:27:00.096601 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d28db16c9f7e405b8866f0e6c548e246. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:27:00.097977 23828 consensus_queue.cc:1035] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:27:00.124436 23493 raft_consensus.cc:2932] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:00.129226 19093 raft_consensus.cc:2932] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:00.132362 18832 raft_consensus.cc:2932] T d28db16c9f7e405b8866f0e6c548e246 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:00.134478 18553 catalog_manager.cc:5484] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b reported cstate change: config changed from index -1 to 4, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:00.280205 19076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:27:00.308161 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:00.348870 20894 raft_consensus.cc:1203] T 246a8425400f4890a5264c4ae8a627c5 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:27:00.355454 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:00.401844 23917 ts_tablet_manager.cc:902] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681: Initiating tablet copy from peer 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:27:00.426268 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:00.429711 23920 ts_tablet_manager.cc:902] T 51ef0be93c914e258635e13f3cca79aa P ff6acba046404ef0a6f41ad5e09d852f: Initiating tablet copy from peer 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:27:00.431306 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:00.460309 21007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:00.460353 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:00.469568 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:00.523629 23920 tablet_copy_client.cc:308] T 51ef0be93c914e258635e13f3cca79aa P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.65:36749
I20240502 14:27:00.524307 18721 tablet_copy_service.cc:140] P 2db304b538e14857b7b1cc272d0db94b: Received BeginTabletCopySession request for tablet 51ef0be93c914e258635e13f3cca79aa from peer ff6acba046404ef0a6f41ad5e09d852f ({username='slave'} at 127.5.107.72:37745)
I20240502 14:27:00.524518 18721 tablet_copy_service.cc:161] P 2db304b538e14857b7b1cc272d0db94b: Beginning new tablet copy session on tablet 51ef0be93c914e258635e13f3cca79aa from peer ff6acba046404ef0a6f41ad5e09d852f at {username='slave'} at 127.5.107.72:37745: session id = ff6acba046404ef0a6f41ad5e09d852f-51ef0be93c914e258635e13f3cca79aa
I20240502 14:27:00.527550 18721 tablet_copy_source_session.cc:205] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:27:00.530061 23920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51ef0be93c914e258635e13f3cca79aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:27:00.534953 23917 tablet_copy_client.cc:308] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.65:36749
I20240502 14:27:00.535774 18721 tablet_copy_service.cc:140] P 2db304b538e14857b7b1cc272d0db94b: Received BeginTabletCopySession request for tablet 51ef0be93c914e258635e13f3cca79aa from peer a834edffbcb2469aae598b069aab0681 ({username='slave'} at 127.5.107.69:53895)
I20240502 14:27:00.535945 18721 tablet_copy_service.cc:161] P 2db304b538e14857b7b1cc272d0db94b: Beginning new tablet copy session on tablet 51ef0be93c914e258635e13f3cca79aa from peer a834edffbcb2469aae598b069aab0681 at {username='slave'} at 127.5.107.69:53895: session id = a834edffbcb2469aae598b069aab0681-51ef0be93c914e258635e13f3cca79aa
I20240502 14:27:00.538726 18721 tablet_copy_source_session.cc:205] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:27:00.540927 23917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51ef0be93c914e258635e13f3cca79aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:27:00.556119 23939 ts_tablet_manager.cc:902] T d28db16c9f7e405b8866f0e6c548e246 P a834edffbcb2469aae598b069aab0681: Initiating tablet copy from peer 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:27:00.589092 23917 tablet_copy_client.cc:783] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681: tablet copy: Starting download of 0 data blocks...
I20240502 14:27:00.589968 23917 tablet_copy_client.cc:647] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681: tablet copy: Starting download of 1 WAL segments...
I20240502 14:27:00.592979 23920 tablet_copy_client.cc:783] T 51ef0be93c914e258635e13f3cca79aa P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Starting download of 0 data blocks...
I20240502 14:27:00.593288 23920 tablet_copy_client.cc:647] T 51ef0be93c914e258635e13f3cca79aa P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Starting download of 1 WAL segments...
I20240502 14:27:00.603997 23920 tablet_copy_client.cc:523] T 51ef0be93c914e258635e13f3cca79aa P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:27:00.606228 23917 tablet_copy_client.cc:523] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:27:00.612080 23939 tablet_copy_client.cc:308] T d28db16c9f7e405b8866f0e6c548e246 P a834edffbcb2469aae598b069aab0681: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.65:36749
I20240502 14:27:00.612834 18721 tablet_copy_service.cc:140] P 2db304b538e14857b7b1cc272d0db94b: Received BeginTabletCopySession request for tablet d28db16c9f7e405b8866f0e6c548e246 from peer a834edffbcb2469aae598b069aab0681 ({username='slave'} at 127.5.107.69:53895)
I20240502 14:27:00.613000 18721 tablet_copy_service.cc:161] P 2db304b538e14857b7b1cc272d0db94b: Beginning new tablet copy session on tablet d28db16c9f7e405b8866f0e6c548e246 from peer a834edffbcb2469aae598b069aab0681 at {username='slave'} at 127.5.107.69:53895: session id = a834edffbcb2469aae598b069aab0681-d28db16c9f7e405b8866f0e6c548e246
I20240502 14:27:00.616745 18721 tablet_copy_source_session.cc:205] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:27:00.619915 23939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d28db16c9f7e405b8866f0e6c548e246. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:27:00.627671 23920 tablet_bootstrap.cc:492] T 51ef0be93c914e258635e13f3cca79aa P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap starting.
I20240502 14:27:00.642195 23939 tablet_copy_client.cc:783] T d28db16c9f7e405b8866f0e6c548e246 P a834edffbcb2469aae598b069aab0681: tablet copy: Starting download of 0 data blocks...
I20240502 14:27:00.643532 23939 tablet_copy_client.cc:647] T d28db16c9f7e405b8866f0e6c548e246 P a834edffbcb2469aae598b069aab0681: tablet copy: Starting download of 1 WAL segments...
I20240502 14:27:00.644048 23917 tablet_bootstrap.cc:492] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681: Bootstrap starting.
I20240502 14:27:00.646709 23939 tablet_copy_client.cc:523] T d28db16c9f7e405b8866f0e6c548e246 P a834edffbcb2469aae598b069aab0681: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:27:00.673651 23939 tablet_bootstrap.cc:492] T d28db16c9f7e405b8866f0e6c548e246 P a834edffbcb2469aae598b069aab0681: Bootstrap starting.
I20240502 14:27:00.858956 23917 tablet_bootstrap.cc:492] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:27:00.859561 23917 tablet_bootstrap.cc:492] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681: Bootstrap complete.
I20240502 14:27:00.859898 23917 ts_tablet_manager.cc:1367] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681: Time spent bootstrapping tablet: real 0.216s user 0.063s sys 0.000s
I20240502 14:27:00.860819 23917 raft_consensus.cc:348] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:00.861169 23917 raft_consensus.cc:729] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a834edffbcb2469aae598b069aab0681, State: Initialized, Role: FOLLOWER
I20240502 14:27:00.861549 23917 consensus_queue.cc:260] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:00.868933 23917 ts_tablet_manager.cc:1398] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20240502 14:27:00.870937 18721 tablet_copy_service.cc:342] P 2db304b538e14857b7b1cc272d0db94b: Request end of tablet copy session a834edffbcb2469aae598b069aab0681-51ef0be93c914e258635e13f3cca79aa received from {username='slave'} at 127.5.107.69:53895
I20240502 14:27:00.871131 18721 tablet_copy_service.cc:433] P 2db304b538e14857b7b1cc272d0db94b: ending tablet copy session a834edffbcb2469aae598b069aab0681-51ef0be93c914e258635e13f3cca79aa on tablet 51ef0be93c914e258635e13f3cca79aa with peer a834edffbcb2469aae598b069aab0681
I20240502 14:27:00.882985 23920 tablet_bootstrap.cc:492] T 51ef0be93c914e258635e13f3cca79aa P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:27:00.883613 23920 tablet_bootstrap.cc:492] T 51ef0be93c914e258635e13f3cca79aa P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap complete.
I20240502 14:27:00.884140 23920 ts_tablet_manager.cc:1367] T 51ef0be93c914e258635e13f3cca79aa P ff6acba046404ef0a6f41ad5e09d852f: Time spent bootstrapping tablet: real 0.257s user 0.063s sys 0.002s
I20240502 14:27:00.885066 23920 raft_consensus.cc:348] T 51ef0be93c914e258635e13f3cca79aa P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:00.885373 23920 raft_consensus.cc:729] T 51ef0be93c914e258635e13f3cca79aa P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6acba046404ef0a6f41ad5e09d852f, State: Initialized, Role: FOLLOWER
I20240502 14:27:00.885689 23920 consensus_queue.cc:260] T 51ef0be93c914e258635e13f3cca79aa P ff6acba046404ef0a6f41ad5e09d852f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:00.892961 23939 tablet_bootstrap.cc:492] T d28db16c9f7e405b8866f0e6c548e246 P a834edffbcb2469aae598b069aab0681: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:27:00.893661 23939 tablet_bootstrap.cc:492] T d28db16c9f7e405b8866f0e6c548e246 P a834edffbcb2469aae598b069aab0681: Bootstrap complete.
I20240502 14:27:00.894073 23939 ts_tablet_manager.cc:1367] T d28db16c9f7e405b8866f0e6c548e246 P a834edffbcb2469aae598b069aab0681: Time spent bootstrapping tablet: real 0.221s user 0.057s sys 0.006s
I20240502 14:27:00.895190 23939 raft_consensus.cc:348] T d28db16c9f7e405b8866f0e6c548e246 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:27:00.895684 23939 raft_consensus.cc:729] T d28db16c9f7e405b8866f0e6c548e246 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a834edffbcb2469aae598b069aab0681, State: Initialized, Role: FOLLOWER
I20240502 14:27:00.896106 23939 consensus_queue.cc:260] T d28db16c9f7e405b8866f0e6c548e246 P a834edffbcb2469aae598b069aab0681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:27:00.898337 20627 raft_consensus.cc:1203] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:27:00.899372 23939 ts_tablet_manager.cc:1398] T d28db16c9f7e405b8866f0e6c548e246 P a834edffbcb2469aae598b069aab0681: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:27:00.901476 18721 tablet_copy_service.cc:342] P 2db304b538e14857b7b1cc272d0db94b: Request end of tablet copy session a834edffbcb2469aae598b069aab0681-d28db16c9f7e405b8866f0e6c548e246 received from {username='slave'} at 127.5.107.69:53895
I20240502 14:27:00.901619 18721 tablet_copy_service.cc:433] P 2db304b538e14857b7b1cc272d0db94b: ending tablet copy session a834edffbcb2469aae598b069aab0681-d28db16c9f7e405b8866f0e6c548e246 on tablet d28db16c9f7e405b8866f0e6c548e246 with peer a834edffbcb2469aae598b069aab0681
I20240502 14:27:01.013428 23920 ts_tablet_manager.cc:1398] T 51ef0be93c914e258635e13f3cca79aa P ff6acba046404ef0a6f41ad5e09d852f: Time spent starting tablet: real 0.129s user 0.027s sys 0.022s
I20240502 14:27:01.032269 19076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:27:01.067368 20627 raft_consensus.cc:1203] T d28db16c9f7e405b8866f0e6c548e246 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:27:01.067957 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:01.072487 21028 raft_consensus.cc:1203] T 51ef0be93c914e258635e13f3cca79aa P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:27:01.074167 18721 tablet_copy_service.cc:342] P 2db304b538e14857b7b1cc272d0db94b: Request end of tablet copy session ff6acba046404ef0a6f41ad5e09d852f-51ef0be93c914e258635e13f3cca79aa received from {username='slave'} at 127.5.107.72:37745
I20240502 14:27:01.074313 18721 tablet_copy_service.cc:433] P 2db304b538e14857b7b1cc272d0db94b: ending tablet copy session ff6acba046404ef0a6f41ad5e09d852f-51ef0be93c914e258635e13f3cca79aa on tablet 51ef0be93c914e258635e13f3cca79aa with peer ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:27:01.087827 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:01.098650 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:01.111411 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:01.113440 21007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:01.138808 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:01.192831 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:01.306175 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:01.337785 19076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:27:01.338485 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:01.359282 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:01.422750 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:01.441651 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:01.450436 19076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:27:01.452900 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:01.502349 21007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:01.525997 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:01.534847 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:01.539211 19076 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:27:01.545444 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:01.578264 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:01.603075 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:01.611508 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:01.617645 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:01.628191 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:01.639487 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:01.685388 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:01.697722 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:01.754601 21007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:01.784258 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:01.804445 21007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:01.850567 19097 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1ecc6539b85d40e49df7ffc5f93cb801"
dest_uuid: "007b2e79dc424a6f80141180f072f2d8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48570
I20240502 14:27:01.850925 19097 raft_consensus.cc:595] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:27:02.003376 24107 raft_consensus.cc:979] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8: : Instructing follower ff6acba046404ef0a6f41ad5e09d852f to start an election
I20240502 14:27:02.003564 24107 raft_consensus.cc:1067] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Signalling peer ff6acba046404ef0a6f41ad5e09d852f to start an election
I20240502 14:27:02.004725 21028 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1ecc6539b85d40e49df7ffc5f93cb801"
dest_uuid: "ff6acba046404ef0a6f41ad5e09d852f"
from {username='slave'} at 127.5.107.68:40869
I20240502 14:27:02.005143 21028 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:27:02.005654 21028 raft_consensus.cc:3037] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:27:02.009971 21028 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:02.012823 18700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ecc6539b85d40e49df7ffc5f93cb801" candidate_uuid: "ff6acba046404ef0a6f41ad5e09d852f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "2db304b538e14857b7b1cc272d0db94b"
I20240502 14:27:02.013108 18700 raft_consensus.cc:3037] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:27:02.014375 21028 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f [CANDIDATE]: Term 2 election: Requested vote from peers 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749), c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409), 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:27:02.016829 18700 raft_consensus.cc:2454] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff6acba046404ef0a6f41ad5e09d852f in term 2.
I20240502 14:27:02.019042 19097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ecc6539b85d40e49df7ffc5f93cb801" candidate_uuid: "ff6acba046404ef0a6f41ad5e09d852f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "007b2e79dc424a6f80141180f072f2d8"
I20240502 14:27:02.019331 19097 raft_consensus.cc:3032] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:27:02.019443 19097 raft_consensus.cc:729] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Running, Role: LEADER
I20240502 14:27:02.019769 19097 consensus_queue.cc:260] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:02.020234 19097 raft_consensus.cc:3037] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:27:02.023391 19097 raft_consensus.cc:2454] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff6acba046404ef0a6f41ad5e09d852f in term 2.
I20240502 14:27:02.039687 20964 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 007b2e79dc424a6f80141180f072f2d8, 2db304b538e14857b7b1cc272d0db94b, ff6acba046404ef0a6f41ad5e09d852f; no voters:
I20240502 14:27:02.098013 24141 raft_consensus.cc:2781] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f [term 2 FOLLOWER]: Leader election won for term 2
W20240502 14:27:02.098098 20964 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.5.107.68:41463, user_credentials={real_user=slave}} blocked reactor thread for 58468.2us
I20240502 14:27:02.145299 18963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ecc6539b85d40e49df7ffc5f93cb801" candidate_uuid: "ff6acba046404ef0a6f41ad5e09d852f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c902805060df4df9b5599d0e94b2dc06"
I20240502 14:27:02.145653 18963 raft_consensus.cc:3037] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:27:02.149765 18963 raft_consensus.cc:2454] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff6acba046404ef0a6f41ad5e09d852f in term 2.
I20240502 14:27:02.177313 24141 raft_consensus.cc:686] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f [term 2 LEADER]: Becoming Leader. State: Replica: ff6acba046404ef0a6f41ad5e09d852f, State: Running, Role: LEADER
I20240502 14:27:02.178113 24141 consensus_queue.cc:237] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:02.195521 18551 catalog_manager.cc:5484] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f reported cstate change: term changed from 1 to 2, leader changed from 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) to ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72). New cstate: current_term: 2 leader_uuid: "ff6acba046404ef0a6f41ad5e09d852f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:02.404044 18700 consensus_queue.cc:237] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:02.407393 21028 raft_consensus.cc:1261] T 51ef0be93c914e258635e13f3cca79aa P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:02.408277 24098 consensus_queue.cc:1035] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:02.413097 20627 raft_consensus.cc:1261] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:02.413920 18963 raft_consensus.cc:1261] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:02.415393 24098 consensus_queue.cc:1035] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:02.416227 24098 consensus_queue.cc:1035] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:02.460047 19076 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:02.462002 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:02.517673 24098 raft_consensus.cc:2932] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:02.517949 21029 raft_consensus.cc:2932] T 51ef0be93c914e258635e13f3cca79aa P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:02.519316 19076 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:02.530313 18963 raft_consensus.cc:2932] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:02.541690 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:02.552882 18556 catalog_manager.cc:5484] T 51ef0be93c914e258635e13f3cca79aa P ff6acba046404ef0a6f41ad5e09d852f reported cstate change: config changed from index 5 to 6, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:02.561597 18812 tablet_service.cc:1514] Processing DeleteTablet for tablet 51ef0be93c914e258635e13f3cca79aa with delete_type TABLET_DATA_TOMBSTONED (TS fec04c50dc5b49878ffa3c3873df32e4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60970
I20240502 14:27:02.591648 24244 tablet_replica.cc:330] T 51ef0be93c914e258635e13f3cca79aa P fec04c50dc5b49878ffa3c3873df32e4: stopping tablet replica
I20240502 14:27:02.592485 18965 raft_consensus.cc:1261] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06 [term 2 FOLLOWER]: Refusing update from remote peer ff6acba046404ef0a6f41ad5e09d852f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:27:02.592909 24244 raft_consensus.cc:2229] T 51ef0be93c914e258635e13f3cca79aa P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:02.593513 24141 consensus_queue.cc:1035] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:02.594877 24244 raft_consensus.cc:2258] T 51ef0be93c914e258635e13f3cca79aa P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:02.603704 24244 ts_tablet_manager.cc:1875] T 51ef0be93c914e258635e13f3cca79aa P fec04c50dc5b49878ffa3c3873df32e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:02.626653 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:02.653439 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:02.685487 18700 raft_consensus.cc:1261] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Refusing update from remote peer ff6acba046404ef0a6f41ad5e09d852f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:27:02.685412 20627 raft_consensus.cc:2932] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:02.686691 24141 consensus_queue.cc:1035] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:02.703116 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:02.726886 24244 ts_tablet_manager.cc:1888] T 51ef0be93c914e258635e13f3cca79aa P fec04c50dc5b49878ffa3c3873df32e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:27:02.774425 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:02.798234 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:02.814924 24244 log.cc:1196] T 51ef0be93c914e258635e13f3cca79aa P fec04c50dc5b49878ffa3c3873df32e4: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/wals/51ef0be93c914e258635e13f3cca79aa
I20240502 14:27:02.822978 18543 catalog_manager.cc:4830] TS fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009): tablet 51ef0be93c914e258635e13f3cca79aa (table rebalance_test_table_8 [id=5a8a0451cdf0479c94edf1d94e77fb83]) successfully deleted
I20240502 14:27:02.825433 19093 raft_consensus.cc:1261] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [term 2 FOLLOWER]: Refusing update from remote peer ff6acba046404ef0a6f41ad5e09d852f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:27:02.826730 24141 consensus_queue.cc:1035] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:02.847718 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:02.861691 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:02.884955 21007 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:02.897125 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:02.908447 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:27:02.911667 20962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.65:36749, user_credentials={real_user=slave}} blocked reactor thread for 58856.8us
I20240502 14:27:02.932919 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:03.066344 21007 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:03.510957 19076 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:03.575776 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:03.598793 18701 consensus_queue.cc:237] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:03.600677 19076 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:03.625985 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:03.640583 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:03.640971 21028 raft_consensus.cc:1261] T db56136f2dbb429098d4d78b15a3a439 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:03.641955 24228 consensus_queue.cc:1035] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:03.645210 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:03.647905 20627 raft_consensus.cc:1261] T db56136f2dbb429098d4d78b15a3a439 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:03.648730 24228 consensus_queue.cc:1035] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:03.670382 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:03.684415 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:03.703888 19093 raft_consensus.cc:1261] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:03.706429 24228 consensus_queue.cc:1035] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:03.712404 19076 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:03.733392 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 14:27:03.734597 24400 log.cc:924] Time spent T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
I20240502 14:27:03.746866 24241 raft_consensus.cc:2932] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:03.749830 20627 raft_consensus.cc:2932] T db56136f2dbb429098d4d78b15a3a439 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:03.750277 20607 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:03.750638 20606 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:03.768644 21028 raft_consensus.cc:2932] T db56136f2dbb429098d4d78b15a3a439 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:03.784528 19093 raft_consensus.cc:2932] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:03.802755 18551 catalog_manager.cc:5484] T db56136f2dbb429098d4d78b15a3a439 P ff6acba046404ef0a6f41ad5e09d852f reported cstate change: config changed from index 5 to 6, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:03.811352 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:03.814502 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:03.832350 20606 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:03.848914 21007 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:03.886889 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:03.903879 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:03.914944 18812 tablet_service.cc:1514] Processing DeleteTablet for tablet db56136f2dbb429098d4d78b15a3a439 with delete_type TABLET_DATA_TOMBSTONED (TS fec04c50dc5b49878ffa3c3873df32e4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60970
I20240502 14:27:03.920104 21007 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:03.923149 18813 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:27:03.961222 18900 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.69:41119, user_credentials={real_user=slave}} blocked reactor thread for 48913.6us
I20240502 14:27:03.960884 24440 tablet_replica.cc:330] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4: stopping tablet replica
I20240502 14:27:03.965616 24440 raft_consensus.cc:2229] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:03.966024 24440 raft_consensus.cc:2258] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:03.966892 24440 ts_tablet_manager.cc:1875] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:03.996822 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:04.014583 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:04.082080 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:04.098960 24440 ts_tablet_manager.cc:1888] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:27:04.099249 24440 log.cc:1196] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/wals/db56136f2dbb429098d4d78b15a3a439
I20240502 14:27:04.124516 18543 catalog_manager.cc:4830] TS fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009): tablet db56136f2dbb429098d4d78b15a3a439 (table rebalance_test_table_8 [id=5a8a0451cdf0479c94edf1d94e77fb83]) successfully deleted
I20240502 14:27:04.195395 21007 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:04.586443 19076 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:04.665064 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:04.677783 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:04.759088 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:04.776388 20606 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:04.796025 18833 consensus_queue.cc:237] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:04.800684 21007 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:04.804737 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:04.813930 18701 consensus_queue.cc:237] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:27:04.845619 20626 raft_consensus.cc:1261] T d28db16c9f7e405b8866f0e6c548e246 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:27:04.845860 18832 raft_consensus.cc:1261] T d28db16c9f7e405b8866f0e6c548e246 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:27:04.846626 24417 consensus_queue.cc:1035] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:04.847831 24417 consensus_queue.cc:1035] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:04.848177 19093 raft_consensus.cc:1261] T 0d8667d91fd64809b635362c4a8fa33f P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:04.849821 19076 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:04.850438 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:04.850731 24428 consensus_queue.cc:1035] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:04.852247 20894 raft_consensus.cc:1261] T 0d8667d91fd64809b635362c4a8fa33f P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:04.853063 21028 raft_consensus.cc:1261] T 0d8667d91fd64809b635362c4a8fa33f P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:04.857165 23700 consensus_queue.cc:1035] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:04.859941 23700 consensus_queue.cc:1035] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:04.859515 24417 raft_consensus.cc:2932] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:04.862107 18832 raft_consensus.cc:2932] T d28db16c9f7e405b8866f0e6c548e246 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:04.868750 18553 catalog_manager.cc:5484] T d28db16c9f7e405b8866f0e6c548e246 P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index 4 to 5, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:04.869325 20626 raft_consensus.cc:2932] T d28db16c9f7e405b8866f0e6c548e246 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:04.906562 20606 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:04.917090 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:04.936200 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:04.997443 24428 raft_consensus.cc:2932] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:05.008692 19093 raft_consensus.cc:2932] T 0d8667d91fd64809b635362c4a8fa33f P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:05.017295 21029 raft_consensus.cc:2932] T 0d8667d91fd64809b635362c4a8fa33f P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:05.058374 18963 consensus_queue.cc:237] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:27:05.067062 20893 raft_consensus.cc:2932] T 0d8667d91fd64809b635362c4a8fa33f P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:05.075737 20760 raft_consensus.cc:1261] T 21af917743444f2db932636a78f33512 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:27:05.076970 24264 consensus_queue.cc:1035] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:05.085583 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:05.086431 19093 raft_consensus.cc:1261] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:27:05.087512 24264 consensus_queue.cc:1035] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:05.088197 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:05.097518 21007 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:05.115607 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:05.127631 18558 catalog_manager.cc:5484] T 0d8667d91fd64809b635362c4a8fa33f P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: config changed from index 5 to 6, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:05.142084 24264 raft_consensus.cc:2932] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:05.152233 18681 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d8667d91fd64809b635362c4a8fa33f with delete_type TABLET_DATA_TOMBSTONED (TS 2db304b538e14857b7b1cc272d0db94b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54454
I20240502 14:27:05.153896 18560 catalog_manager.cc:5484] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 reported cstate change: config changed from index 4 to 5, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:05.152226 19093 raft_consensus.cc:2932] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:05.156549 20760 raft_consensus.cc:2932] T 21af917743444f2db932636a78f33512 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:05.161063 19076 tablet_service.cc:1514] Processing DeleteTablet for tablet d28db16c9f7e405b8866f0e6c548e246 with delete_type TABLET_DATA_TOMBSTONED (TS 007b2e79dc424a6f80141180f072f2d8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41468
I20240502 14:27:05.184494 18812 tablet_service.cc:1514] Processing DeleteTablet for tablet 21af917743444f2db932636a78f33512 with delete_type TABLET_DATA_TOMBSTONED (TS fec04c50dc5b49878ffa3c3873df32e4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:60970
I20240502 14:27:05.187420 24598 tablet_replica.cc:330] T 21af917743444f2db932636a78f33512 P fec04c50dc5b49878ffa3c3873df32e4: stopping tablet replica
I20240502 14:27:05.187736 24598 raft_consensus.cc:2229] T 21af917743444f2db932636a78f33512 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:05.190317 24598 raft_consensus.cc:2258] T 21af917743444f2db932636a78f33512 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:05.195782 24593 tablet_replica.cc:330] T 0d8667d91fd64809b635362c4a8fa33f P 2db304b538e14857b7b1cc272d0db94b: stopping tablet replica
I20240502 14:27:05.196530 24593 raft_consensus.cc:2229] T 0d8667d91fd64809b635362c4a8fa33f P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:05.197023 24593 raft_consensus.cc:2258] T 0d8667d91fd64809b635362c4a8fa33f P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:05.198163 24598 ts_tablet_manager.cc:1875] T 21af917743444f2db932636a78f33512 P fec04c50dc5b49878ffa3c3873df32e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:05.198995 24593 ts_tablet_manager.cc:1875] T 0d8667d91fd64809b635362c4a8fa33f P 2db304b538e14857b7b1cc272d0db94b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:05.199677 24595 tablet_replica.cc:330] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8: stopping tablet replica
I20240502 14:27:05.200181 24595 raft_consensus.cc:2229] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:05.200690 24595 raft_consensus.cc:2258] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:05.202497 24595 ts_tablet_manager.cc:1875] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:05.244258 24593 ts_tablet_manager.cc:1888] T 0d8667d91fd64809b635362c4a8fa33f P 2db304b538e14857b7b1cc272d0db94b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:27:05.263582 24593 log.cc:1196] T 0d8667d91fd64809b635362c4a8fa33f P 2db304b538e14857b7b1cc272d0db94b: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal/wals/0d8667d91fd64809b635362c4a8fa33f
I20240502 14:27:05.265393 18544 catalog_manager.cc:4830] TS 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749): tablet 0d8667d91fd64809b635362c4a8fa33f (table rebalance_test_table_2 [id=0e84c82d60e64dbd9149c5f9fb7ff3c8]) successfully deleted
I20240502 14:27:05.306985 24595 ts_tablet_manager.cc:1888] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:27:05.307227 24595 log.cc:1196] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal/wals/d28db16c9f7e405b8866f0e6c548e246
I20240502 14:27:05.308326 18546 catalog_manager.cc:4830] TS 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463): tablet d28db16c9f7e405b8866f0e6c548e246 (table rebalance_test_table_0 [id=27066b01c29c4a1380ee4753a1c036d3]) successfully deleted
I20240502 14:27:05.315536 24598 ts_tablet_manager.cc:1888] T 21af917743444f2db932636a78f33512 P fec04c50dc5b49878ffa3c3873df32e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:27:05.315765 24598 log.cc:1196] T 21af917743444f2db932636a78f33512 P fec04c50dc5b49878ffa3c3873df32e4: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/wals/21af917743444f2db932636a78f33512
I20240502 14:27:05.316850 18543 catalog_manager.cc:4830] TS fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009): tablet 21af917743444f2db932636a78f33512 (table rebalance_test_table_8 [id=5a8a0451cdf0479c94edf1d94e77fb83]) successfully deleted
I20240502 14:27:05.409461 18963 consensus_queue.cc:237] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:27:05.423090 20893 raft_consensus.cc:1261] T cf2a6426ae6c438f942d39530da6e3e3 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:27:05.424326 24264 consensus_queue.cc:1035] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:05.425500 18699 raft_consensus.cc:1261] T cf2a6426ae6c438f942d39530da6e3e3 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:27:05.428020 24264 consensus_queue.cc:1035] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:05.448402 24264 raft_consensus.cc:2932] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:05.452128 18699 raft_consensus.cc:2932] T cf2a6426ae6c438f942d39530da6e3e3 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:05.466553 18557 catalog_manager.cc:5484] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 reported cstate change: config changed from index 4 to 5, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:05.469697 20893 raft_consensus.cc:2932] T cf2a6426ae6c438f942d39530da6e3e3 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:05.475888 19076 tablet_service.cc:1514] Processing DeleteTablet for tablet cf2a6426ae6c438f942d39530da6e3e3 with delete_type TABLET_DATA_TOMBSTONED (TS 007b2e79dc424a6f80141180f072f2d8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41468
I20240502 14:27:05.476334 24595 tablet_replica.cc:330] T cf2a6426ae6c438f942d39530da6e3e3 P 007b2e79dc424a6f80141180f072f2d8: stopping tablet replica
I20240502 14:27:05.476536 24595 raft_consensus.cc:2229] T cf2a6426ae6c438f942d39530da6e3e3 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:05.476766 24595 raft_consensus.cc:2258] T cf2a6426ae6c438f942d39530da6e3e3 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:05.477876 24595 ts_tablet_manager.cc:1875] T cf2a6426ae6c438f942d39530da6e3e3 P 007b2e79dc424a6f80141180f072f2d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:05.489086 24595 ts_tablet_manager.cc:1888] T cf2a6426ae6c438f942d39530da6e3e3 P 007b2e79dc424a6f80141180f072f2d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:27:05.489436 24595 log.cc:1196] T cf2a6426ae6c438f942d39530da6e3e3 P 007b2e79dc424a6f80141180f072f2d8: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal/wals/cf2a6426ae6c438f942d39530da6e3e3
I20240502 14:27:05.490623 18546 catalog_manager.cc:4830] TS 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463): tablet cf2a6426ae6c438f942d39530da6e3e3 (table rebalance_test_table_0 [id=27066b01c29c4a1380ee4753a1c036d3]) successfully deleted
I20240502 14:27:05.676741 19093 consensus_queue.cc:237] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:27:05.679816 20760 raft_consensus.cc:1261] T bad03a4ec8494938839d7d6eef8c684e P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:27:05.679951 18699 raft_consensus.cc:1261] T bad03a4ec8494938839d7d6eef8c684e P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:27:05.680857 23704 consensus_queue.cc:1035] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:05.681690 23704 consensus_queue.cc:1035] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:05.712913 19076 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:05.717139 23704 raft_consensus.cc:2932] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:05.722781 20759 raft_consensus.cc:2932] T bad03a4ec8494938839d7d6eef8c684e P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:05.728281 18560 catalog_manager.cc:5484] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: config changed from index 4 to 5, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:05.730397 18701 raft_consensus.cc:2932] T bad03a4ec8494938839d7d6eef8c684e P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:05.763209 18945 tablet_service.cc:1514] Processing DeleteTablet for tablet bad03a4ec8494938839d7d6eef8c684e with delete_type TABLET_DATA_TOMBSTONED (TS c902805060df4df9b5599d0e94b2dc06 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55752
I20240502 14:27:05.768321 24723 tablet_replica.cc:330] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06: stopping tablet replica
I20240502 14:27:05.772138 24723 raft_consensus.cc:2229] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:05.773380 24723 raft_consensus.cc:2258] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:05.785111 24723 ts_tablet_manager.cc:1875] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:05.800839 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:05.813017 24723 ts_tablet_manager.cc:1888] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:27:05.814220 24723 log.cc:1196] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal/wals/bad03a4ec8494938839d7d6eef8c684e
I20240502 14:27:05.816773 18546 catalog_manager.cc:4830] TS c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409): tablet bad03a4ec8494938839d7d6eef8c684e (table rebalance_test_table_3 [id=210510c054044948932a30db816646aa]) successfully deleted
I20240502 14:27:05.880045 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:05.943728 20606 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:05.958063 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:05.962936 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:05.980857 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 14:27:06.018162 18634 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.66:33009, user_credentials={real_user=slave}} blocked reactor thread for 112884us
I20240502 14:27:06.046911 21007 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:06.152287 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:06.181083 19076 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:06.222329 20606 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:06.241879 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:06.248732 21007 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:06.295430 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:06.343870 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:06.364980 18701 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "51ef0be93c914e258635e13f3cca79aa"
dest_uuid: "2db304b538e14857b7b1cc272d0db94b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40442
I20240502 14:27:06.365676 18701 raft_consensus.cc:595] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Received request to transfer leadership
I20240502 14:27:06.390184 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:06.425354 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:06.438704 19076 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:06.458184 21007 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:06.483573 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:06.492794 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:06.495458 19076 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:06.513859 20606 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:06.528609 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:06.550050 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:06.581616 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:06.604612 24756 raft_consensus.cc:979] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b: : Instructing follower a834edffbcb2469aae598b069aab0681 to start an election
I20240502 14:27:06.604815 24756 raft_consensus.cc:1067] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Signalling peer a834edffbcb2469aae598b069aab0681 to start an election
I20240502 14:27:06.606178 20625 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "51ef0be93c914e258635e13f3cca79aa"
dest_uuid: "a834edffbcb2469aae598b069aab0681"
from {username='slave'} at 127.5.107.65:53335
I20240502 14:27:06.606581 20625 raft_consensus.cc:482] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:27:06.606810 20625 raft_consensus.cc:3037] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:27:06.607446 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:06.610486 20625 raft_consensus.cc:504] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:06.616439 18963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ef0be93c914e258635e13f3cca79aa" candidate_uuid: "a834edffbcb2469aae598b069aab0681" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "c902805060df4df9b5599d0e94b2dc06"
I20240502 14:27:06.616684 18963 raft_consensus.cc:3037] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:27:06.620121 18963 raft_consensus.cc:2454] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a834edffbcb2469aae598b069aab0681 in term 2.
I20240502 14:27:06.620347 20606 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:06.621862 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ef0be93c914e258635e13f3cca79aa" candidate_uuid: "a834edffbcb2469aae598b069aab0681" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "2db304b538e14857b7b1cc272d0db94b"
I20240502 14:27:06.622095 18701 raft_consensus.cc:3032] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:27:06.622196 18701 raft_consensus.cc:729] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Running, Role: LEADER
I20240502 14:27:06.622551 18701 consensus_queue.cc:260] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:06.622990 18701 raft_consensus.cc:3037] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:27:06.629716 18701 raft_consensus.cc:2454] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a834edffbcb2469aae598b069aab0681 in term 2.
I20240502 14:27:06.630348 20560 leader_election.cc:302] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b, a834edffbcb2469aae598b069aab0681, c902805060df4df9b5599d0e94b2dc06; no voters:
I20240502 14:27:06.639984 20625 leader_election.cc:288] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 [CANDIDATE]: Term 2 election: Requested vote from peers c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409), 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749), ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871)
I20240502 14:27:06.657920 24850 raft_consensus.cc:2781] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:27:06.673833 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:06.683673 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:06.691825 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:06.706344 24850 raft_consensus.cc:686] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 [term 2 LEADER]: Becoming Leader. State: Replica: a834edffbcb2469aae598b069aab0681, State: Running, Role: LEADER
I20240502 14:27:06.707046 24850 consensus_queue.cc:237] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:06.721671 21007 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:06.726830 18557 catalog_manager.cc:5484] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 reported cstate change: term changed from 1 to 2, leader changed from 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) to a834edffbcb2469aae598b069aab0681 (127.5.107.69). New cstate: current_term: 2 leader_uuid: "a834edffbcb2469aae598b069aab0681" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:06.746134 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ef0be93c914e258635e13f3cca79aa" candidate_uuid: "a834edffbcb2469aae598b069aab0681" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "ff6acba046404ef0a6f41ad5e09d852f"
I20240502 14:27:06.746528 21029 raft_consensus.cc:3037] T 51ef0be93c914e258635e13f3cca79aa P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:27:06.749928 21029 raft_consensus.cc:2454] T 51ef0be93c914e258635e13f3cca79aa P ff6acba046404ef0a6f41ad5e09d852f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a834edffbcb2469aae598b069aab0681 in term 2.
I20240502 14:27:07.018429 19076 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:07.047845 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:07.074370 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:07.076401 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:07.111788 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:07.132440 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:07.171043 18965 raft_consensus.cc:1261] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06 [term 2 FOLLOWER]: Refusing update from remote peer a834edffbcb2469aae598b069aab0681: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:27:07.171559 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:07.172570 24850 consensus_queue.cc:1035] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:07.211549 21007 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:07.239795 21029 raft_consensus.cc:1261] T 51ef0be93c914e258635e13f3cca79aa P ff6acba046404ef0a6f41ad5e09d852f [term 2 FOLLOWER]: Refusing update from remote peer a834edffbcb2469aae598b069aab0681: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:27:07.240458 24850 consensus_queue.cc:1035] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:07.299983 18701 raft_consensus.cc:1261] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Refusing update from remote peer a834edffbcb2469aae598b069aab0681: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:27:07.301478 24850 consensus_queue.cc:1035] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:07.409618 18965 consensus_queue.cc:237] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:27:07.416589 18832 raft_consensus.cc:1261] T b13bcd6e693f4c7687a1aabdafbb83c6 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:27:07.417282 20625 raft_consensus.cc:1261] T b13bcd6e693f4c7687a1aabdafbb83c6 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:27:07.418429 24264 consensus_queue.cc:1035] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:07.419055 24264 consensus_queue.cc:1035] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:07.422412 18833 consensus_queue.cc:237] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:27:07.431720 20626 raft_consensus.cc:1261] T 8a2bed29fcd64f7bbb04286404ef09de P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:27:07.433454 24428 consensus_queue.cc:1035] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:07.436416 18965 raft_consensus.cc:1261] T 8a2bed29fcd64f7bbb04286404ef09de P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:27:07.438074 24577 consensus_queue.cc:1035] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:07.457473 20627 raft_consensus.cc:2932] T b13bcd6e693f4c7687a1aabdafbb83c6 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:07.466847 18833 raft_consensus.cc:2932] T b13bcd6e693f4c7687a1aabdafbb83c6 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:07.475533 18551 catalog_manager.cc:5484] T b13bcd6e693f4c7687a1aabdafbb83c6 P a834edffbcb2469aae598b069aab0681 reported cstate change: config changed from index 4 to 5, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:07.465432 24966 raft_consensus.cc:2932] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:07.497085 24428 raft_consensus.cc:2932] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:07.505023 18965 raft_consensus.cc:2932] T 8a2bed29fcd64f7bbb04286404ef09de P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:07.508872 20627 raft_consensus.cc:2932] T 8a2bed29fcd64f7bbb04286404ef09de P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
W20240502 14:27:07.522939 18900 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.69:41119, user_credentials={real_user=slave}} blocked reactor thread for 56028.6us
I20240502 14:27:07.528028 18681 tablet_service.cc:1514] Processing DeleteTablet for tablet b13bcd6e693f4c7687a1aabdafbb83c6 with delete_type TABLET_DATA_TOMBSTONED (TS 2db304b538e14857b7b1cc272d0db94b not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54454
I20240502 14:27:07.536212 18551 catalog_manager.cc:5484] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index 4 to 5, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:07.544121 19076 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a2bed29fcd64f7bbb04286404ef09de with delete_type TABLET_DATA_TOMBSTONED (TS 007b2e79dc424a6f80141180f072f2d8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41468
I20240502 14:27:07.552598 24977 tablet_replica.cc:330] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b: stopping tablet replica
I20240502 14:27:07.552958 24977 raft_consensus.cc:2229] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:07.553409 24977 raft_consensus.cc:2258] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:07.554145 18699 consensus_queue.cc:237] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:27:07.554399 24977 ts_tablet_manager.cc:1875] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:07.567636 24977 ts_tablet_manager.cc:1888] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:27:07.567963 24977 log.cc:1196] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal/wals/b13bcd6e693f4c7687a1aabdafbb83c6
I20240502 14:27:07.568962 18544 catalog_manager.cc:4830] TS 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749): tablet b13bcd6e693f4c7687a1aabdafbb83c6 (table rebalance_test_table_4 [id=4dd18a179f134d1babc05959227a25cc]) successfully deleted
I20240502 14:27:07.577697 24987 tablet_replica.cc:330] T 8a2bed29fcd64f7bbb04286404ef09de P 007b2e79dc424a6f80141180f072f2d8: stopping tablet replica
I20240502 14:27:07.578078 24987 raft_consensus.cc:2229] T 8a2bed29fcd64f7bbb04286404ef09de P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:07.578414 24987 raft_consensus.cc:2258] T 8a2bed29fcd64f7bbb04286404ef09de P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:07.579486 24987 ts_tablet_manager.cc:1875] T 8a2bed29fcd64f7bbb04286404ef09de P 007b2e79dc424a6f80141180f072f2d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:07.582762 20759 raft_consensus.cc:1261] T b315327141954b53a0f87c2d1dc14b9b P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:27:07.582780 20892 raft_consensus.cc:1261] T b315327141954b53a0f87c2d1dc14b9b P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:27:07.583676 24757 consensus_queue.cc:1035] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:07.584076 24416 consensus_queue.cc:1035] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:07.591967 21028 raft_consensus.cc:1261] T b315327141954b53a0f87c2d1dc14b9b P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:27:07.592576 24757 consensus_queue.cc:1035] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:07.602397 18833 raft_consensus.cc:1261] T b315327141954b53a0f87c2d1dc14b9b P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:27:07.603026 24757 consensus_queue.cc:1035] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:07.621801 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:07.624939 19076 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:07.662371 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:07.662513 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:07.683084 24416 raft_consensus.cc:2932] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:07.689968 21029 raft_consensus.cc:2932] T b315327141954b53a0f87c2d1dc14b9b P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:07.690093 20892 raft_consensus.cc:2932] T b315327141954b53a0f87c2d1dc14b9b P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:07.693833 24987 ts_tablet_manager.cc:1888] T 8a2bed29fcd64f7bbb04286404ef09de P 007b2e79dc424a6f80141180f072f2d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:27:07.693974 24987 log.cc:1196] T 8a2bed29fcd64f7bbb04286404ef09de P 007b2e79dc424a6f80141180f072f2d8: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal/wals/8a2bed29fcd64f7bbb04286404ef09de
I20240502 14:27:07.695376 18546 catalog_manager.cc:4830] TS 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463): tablet 8a2bed29fcd64f7bbb04286404ef09de (table rebalance_test_table_6 [id=995bc008b9f94332bfa2c42a77e41bb2]) successfully deleted
I20240502 14:27:07.697593 18553 catalog_manager.cc:5484] T b315327141954b53a0f87c2d1dc14b9b P ff6acba046404ef0a6f41ad5e09d852f reported cstate change: config changed from index 6 to 7, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:07.699258 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:07.702015 19076 tablet_service.cc:1514] Processing DeleteTablet for tablet b315327141954b53a0f87c2d1dc14b9b with delete_type TABLET_DATA_TOMBSTONED (TS 007b2e79dc424a6f80141180f072f2d8 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:41468
W20240502 14:27:07.712713 18635 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.70:46021, user_credentials={real_user=slave}} blocked reactor thread for 128863us
I20240502 14:27:07.714032 18833 raft_consensus.cc:2932] T b315327141954b53a0f87c2d1dc14b9b P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:07.720101 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:07.739710 20759 raft_consensus.cc:2932] T b315327141954b53a0f87c2d1dc14b9b P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:07.744174 24987 tablet_replica.cc:330] T b315327141954b53a0f87c2d1dc14b9b P 007b2e79dc424a6f80141180f072f2d8: stopping tablet replica
I20240502 14:27:07.744439 24987 raft_consensus.cc:2229] T b315327141954b53a0f87c2d1dc14b9b P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:07.744700 24987 raft_consensus.cc:2258] T b315327141954b53a0f87c2d1dc14b9b P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:07.753255 24987 ts_tablet_manager.cc:1875] T b315327141954b53a0f87c2d1dc14b9b P 007b2e79dc424a6f80141180f072f2d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:07.772382 21007 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:07.812602 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:07.798755 24987 ts_tablet_manager.cc:1888] T b315327141954b53a0f87c2d1dc14b9b P 007b2e79dc424a6f80141180f072f2d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:27:07.849304 24987 log.cc:1196] T b315327141954b53a0f87c2d1dc14b9b P 007b2e79dc424a6f80141180f072f2d8: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal/wals/b315327141954b53a0f87c2d1dc14b9b
I20240502 14:27:07.852118 18546 catalog_manager.cc:4830] TS 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463): tablet b315327141954b53a0f87c2d1dc14b9b (table rebalance_test_table_6 [id=995bc008b9f94332bfa2c42a77e41bb2]) successfully deleted
I20240502 14:27:07.934852 18833 consensus_queue.cc:237] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:27:07.962474 18963 raft_consensus.cc:1261] T 414e03d597934da3a6a74d786aeab134 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:27:07.963490 20892 raft_consensus.cc:1261] T 414e03d597934da3a6a74d786aeab134 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:27:07.964705 24959 consensus_queue.cc:1035] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:07.965317 24577 consensus_queue.cc:1035] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:08.046504 24577 raft_consensus.cc:2932] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:08.052232 18965 raft_consensus.cc:2932] T 414e03d597934da3a6a74d786aeab134 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:08.057963 18553 catalog_manager.cc:5484] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index 4 to 5, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:08.064810 20892 raft_consensus.cc:2932] T 414e03d597934da3a6a74d786aeab134 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:08.068060 19076 tablet_service.cc:1514] Processing DeleteTablet for tablet 414e03d597934da3a6a74d786aeab134 with delete_type TABLET_DATA_TOMBSTONED (TS 007b2e79dc424a6f80141180f072f2d8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41468
I20240502 14:27:08.068576 24987 tablet_replica.cc:330] T 414e03d597934da3a6a74d786aeab134 P 007b2e79dc424a6f80141180f072f2d8: stopping tablet replica
I20240502 14:27:08.068786 24987 raft_consensus.cc:2229] T 414e03d597934da3a6a74d786aeab134 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:08.069103 24987 raft_consensus.cc:2258] T 414e03d597934da3a6a74d786aeab134 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:08.070173 24987 ts_tablet_manager.cc:1875] T 414e03d597934da3a6a74d786aeab134 P 007b2e79dc424a6f80141180f072f2d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:08.079941 24987 ts_tablet_manager.cc:1888] T 414e03d597934da3a6a74d786aeab134 P 007b2e79dc424a6f80141180f072f2d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:27:08.088227 24987 log.cc:1196] T 414e03d597934da3a6a74d786aeab134 P 007b2e79dc424a6f80141180f072f2d8: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal/wals/414e03d597934da3a6a74d786aeab134
I20240502 14:27:08.089624 18546 catalog_manager.cc:4830] TS 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463): tablet 414e03d597934da3a6a74d786aeab134 (table rebalance_test_table_4 [id=4dd18a179f134d1babc05959227a25cc]) successfully deleted
I20240502 14:27:08.332036 20627 consensus_queue.cc:237] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:08.350737 21028 raft_consensus.cc:1261] T 51ef0be93c914e258635e13f3cca79aa P ff6acba046404ef0a6f41ad5e09d852f [term 2 FOLLOWER]: Refusing update from remote peer a834edffbcb2469aae598b069aab0681: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:27:08.352880 24933 consensus_queue.cc:1035] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:27:08.358651 18965 raft_consensus.cc:1261] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06 [term 2 FOLLOWER]: Refusing update from remote peer a834edffbcb2469aae598b069aab0681: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:27:08.359455 24933 consensus_queue.cc:1035] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:27:08.403234 25150 raft_consensus.cc:2932] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:08.405061 18963 raft_consensus.cc:2932] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:08.405417 21028 raft_consensus.cc:2932] T 51ef0be93c914e258635e13f3cca79aa P ff6acba046404ef0a6f41ad5e09d852f [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:08.413753 18551 catalog_manager.cc:5484] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 reported cstate change: config changed from index 6 to 8, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New cstate: current_term: 2 leader_uuid: "a834edffbcb2469aae598b069aab0681" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:08.436729 18681 tablet_service.cc:1514] Processing DeleteTablet for tablet 51ef0be93c914e258635e13f3cca79aa with delete_type TABLET_DATA_TOMBSTONED (TS 2db304b538e14857b7b1cc272d0db94b not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:54454
I20240502 14:27:08.440193 25155 tablet_replica.cc:330] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b: stopping tablet replica
I20240502 14:27:08.441280 25155 raft_consensus.cc:2229] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:08.442193 25155 raft_consensus.cc:2258] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:08.443742 25155 ts_tablet_manager.cc:1875] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:08.457079 25155 ts_tablet_manager.cc:1888] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 14:27:08.457675 25155 log.cc:1196] T 51ef0be93c914e258635e13f3cca79aa P 2db304b538e14857b7b1cc272d0db94b: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal/wals/51ef0be93c914e258635e13f3cca79aa
I20240502 14:27:08.459909 18544 catalog_manager.cc:4830] TS 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749): tablet 51ef0be93c914e258635e13f3cca79aa (table rebalance_test_table_8 [id=5a8a0451cdf0479c94edf1d94e77fb83]) successfully deleted
I20240502 14:27:08.580437 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:08.587802 19076 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:08.628736 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 14:27:08.637187 20964 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.67:45409, user_credentials={real_user=slave}} blocked reactor thread for 58475.8us
I20240502 14:27:08.649920 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:08.656159 19076 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:08.690297 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:08.696163 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:08.706701 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:08.732272 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:08.785427 21007 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:08.839308 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:08.843955 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:08.871526 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:08.912098 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:08.926470 19076 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:08.948812 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:08.951912 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:08.970906 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:09.003139 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:09.031466 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:09.086215 21007 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:09.103271 19076 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:09.124724 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:09.127590 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:09.133014 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:09.142562 21007 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:09.185851 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:09.193873 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:09.240284 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:09.265074 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:09.277482 21007 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:09.302717 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:09.666401 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:09.682871 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:09.692183 21007 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:09.699748 18833 consensus_queue.cc:237] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:27:09.716254 19076 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:09.734160 20759 raft_consensus.cc:1261] T 67827c0262724897a05bf80df579bbf3 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:09.734953 25048 consensus_queue.cc:1035] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:09.747987 20892 raft_consensus.cc:1261] T 67827c0262724897a05bf80df579bbf3 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:09.751235 25048 consensus_queue.cc:1035] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:09.822491 18963 raft_consensus.cc:1261] T 67827c0262724897a05bf80df579bbf3 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:09.823462 25048 consensus_queue.cc:1035] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 14:27:09.857942 25314 log.cc:924] Time spent T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4: Append to log took a long time: real 0.125s user 0.000s sys 0.000s
I20240502 14:27:09.858609 25048 raft_consensus.cc:2932] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:09.869869 18963 raft_consensus.cc:2932] T 67827c0262724897a05bf80df579bbf3 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:09.875296 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:09.902400 18551 catalog_manager.cc:5484] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index 5 to 6, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:09.913607 20892 raft_consensus.cc:2932] T 67827c0262724897a05bf80df579bbf3 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:09.919492 18681 tablet_service.cc:1514] Processing DeleteTablet for tablet 67827c0262724897a05bf80df579bbf3 with delete_type TABLET_DATA_TOMBSTONED (TS 2db304b538e14857b7b1cc272d0db94b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54454
I20240502 14:27:09.922308 18833 consensus_queue.cc:237] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:09.925573 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:09.928309 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:09.928431 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:09.928678 25330 tablet_replica.cc:330] T 67827c0262724897a05bf80df579bbf3 P 2db304b538e14857b7b1cc272d0db94b: stopping tablet replica
I20240502 14:27:09.928946 25330 raft_consensus.cc:2229] T 67827c0262724897a05bf80df579bbf3 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:09.930789 20759 raft_consensus.cc:2932] T 67827c0262724897a05bf80df579bbf3 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:09.932385 20894 raft_consensus.cc:1261] T 0d8667d91fd64809b635362c4a8fa33f P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:27:09.932396 21028 raft_consensus.cc:1261] T 0d8667d91fd64809b635362c4a8fa33f P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:27:09.933054 24959 consensus_queue.cc:1035] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:09.934259 25048 consensus_queue.cc:1035] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:09.938650 25330 raft_consensus.cc:2258] T 67827c0262724897a05bf80df579bbf3 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:09.940165 18699 consensus_queue.cc:237] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:27:09.956552 25330 ts_tablet_manager.cc:1875] T 67827c0262724897a05bf80df579bbf3 P 2db304b538e14857b7b1cc272d0db94b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:09.962937 20627 raft_consensus.cc:1261] T df5f14206a1d46e4866f34fa234f945b P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:27:09.963804 24416 consensus_queue.cc:1035] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:09.970027 19093 raft_consensus.cc:1261] T df5f14206a1d46e4866f34fa234f945b P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:27:09.970839 24416 consensus_queue.cc:1035] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:09.980823 25275 raft_consensus.cc:2932] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:09.986164 21029 raft_consensus.cc:2932] T 0d8667d91fd64809b635362c4a8fa33f P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:09.987742 18551 catalog_manager.cc:5484] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index 6 to 7, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:09.992667 19076 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d8667d91fd64809b635362c4a8fa33f with delete_type TABLET_DATA_TOMBSTONED (TS 007b2e79dc424a6f80141180f072f2d8 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:41468
I20240502 14:27:09.999449 24416 raft_consensus.cc:2932] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:10.003274 19097 raft_consensus.cc:2932] T df5f14206a1d46e4866f34fa234f945b P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:10.010263 18551 catalog_manager.cc:5484] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b reported cstate change: config changed from index 4 to 5, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:10.014792 18945 tablet_service.cc:1514] Processing DeleteTablet for tablet df5f14206a1d46e4866f34fa234f945b with delete_type TABLET_DATA_TOMBSTONED (TS c902805060df4df9b5599d0e94b2dc06 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55752
I20240502 14:27:10.040784 20626 raft_consensus.cc:2932] T df5f14206a1d46e4866f34fa234f945b P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:10.042974 25344 tablet_replica.cc:330] T 0d8667d91fd64809b635362c4a8fa33f P 007b2e79dc424a6f80141180f072f2d8: stopping tablet replica
I20240502 14:27:10.045297 25344 raft_consensus.cc:2229] T 0d8667d91fd64809b635362c4a8fa33f P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:10.048049 20892 raft_consensus.cc:2932] T 0d8667d91fd64809b635362c4a8fa33f P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:10.048961 25344 raft_consensus.cc:2258] T 0d8667d91fd64809b635362c4a8fa33f P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:10.055763 25344 ts_tablet_manager.cc:1875] T 0d8667d91fd64809b635362c4a8fa33f P 007b2e79dc424a6f80141180f072f2d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:10.056102 25346 tablet_replica.cc:330] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06: stopping tablet replica
I20240502 14:27:10.056970 25346 raft_consensus.cc:2229] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:10.059525 25346 raft_consensus.cc:2258] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:10.066124 25330 ts_tablet_manager.cc:1888] T 67827c0262724897a05bf80df579bbf3 P 2db304b538e14857b7b1cc272d0db94b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:27:10.066359 25330 log.cc:1196] T 67827c0262724897a05bf80df579bbf3 P 2db304b538e14857b7b1cc272d0db94b: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal/wals/67827c0262724897a05bf80df579bbf3
I20240502 14:27:10.067405 18544 catalog_manager.cc:4830] TS 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749): tablet 67827c0262724897a05bf80df579bbf3 (table rebalance_test_table_5 [id=ff5cff7e35c746a5bc64cc06f18bf7d9]) successfully deleted
I20240502 14:27:10.062510 25346 ts_tablet_manager.cc:1875] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:10.117476 25344 ts_tablet_manager.cc:1888] T 0d8667d91fd64809b635362c4a8fa33f P 007b2e79dc424a6f80141180f072f2d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:27:10.129223 25346 ts_tablet_manager.cc:1888] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:27:10.129881 25346 log.cc:1196] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal/wals/df5f14206a1d46e4866f34fa234f945b
I20240502 14:27:10.134150 18546 catalog_manager.cc:4830] TS c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409): tablet df5f14206a1d46e4866f34fa234f945b (table rebalance_test_table_7 [id=b260896c8b8744c0a1b4f691850aec93]) successfully deleted
I20240502 14:27:10.134498 25344 log.cc:1196] T 0d8667d91fd64809b635362c4a8fa33f P 007b2e79dc424a6f80141180f072f2d8: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal/wals/0d8667d91fd64809b635362c4a8fa33f
I20240502 14:27:10.135507 18546 catalog_manager.cc:4830] TS 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463): tablet 0d8667d91fd64809b635362c4a8fa33f (table rebalance_test_table_2 [id=0e84c82d60e64dbd9149c5f9fb7ff3c8]) successfully deleted
I20240502 14:27:10.243482 19097 consensus_queue.cc:237] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:27:10.257047 20894 raft_consensus.cc:1261] T 8937c809610542d88fdde3daf002c890 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:27:10.259414 24239 consensus_queue.cc:1035] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:10.265630 20626 raft_consensus.cc:1261] T 8937c809610542d88fdde3daf002c890 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:27:10.266307 24239 consensus_queue.cc:1035] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:10.284078 21028 raft_consensus.cc:1261] T 8937c809610542d88fdde3daf002c890 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:27:10.284204 18833 raft_consensus.cc:1261] T 8937c809610542d88fdde3daf002c890 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:27:10.285323 23704 consensus_queue.cc:1035] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:10.295405 23704 consensus_queue.cc:1035] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:10.300812 23704 raft_consensus.cc:2932] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:10.306927 20894 raft_consensus.cc:2932] T 8937c809610542d88fdde3daf002c890 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:10.309442 18833 raft_consensus.cc:2932] T 8937c809610542d88fdde3daf002c890 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:10.317086 21028 raft_consensus.cc:2932] T 8937c809610542d88fdde3daf002c890 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:10.326282 20626 raft_consensus.cc:2932] T 8937c809610542d88fdde3daf002c890 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:10.330201 18553 catalog_manager.cc:5484] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: config changed from index 6 to 7, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:10.340363 18945 tablet_service.cc:1514] Processing DeleteTablet for tablet 8937c809610542d88fdde3daf002c890 with delete_type TABLET_DATA_TOMBSTONED (TS c902805060df4df9b5599d0e94b2dc06 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55752
I20240502 14:27:10.340917 25346 tablet_replica.cc:330] T 8937c809610542d88fdde3daf002c890 P c902805060df4df9b5599d0e94b2dc06: stopping tablet replica
I20240502 14:27:10.341089 25346 raft_consensus.cc:2229] T 8937c809610542d88fdde3daf002c890 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:10.341358 25346 raft_consensus.cc:2258] T 8937c809610542d88fdde3daf002c890 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:10.342108 25346 ts_tablet_manager.cc:1875] T 8937c809610542d88fdde3daf002c890 P c902805060df4df9b5599d0e94b2dc06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:10.351771 25346 ts_tablet_manager.cc:1888] T 8937c809610542d88fdde3daf002c890 P c902805060df4df9b5599d0e94b2dc06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:27:10.353617 25346 log.cc:1196] T 8937c809610542d88fdde3daf002c890 P c902805060df4df9b5599d0e94b2dc06: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal/wals/8937c809610542d88fdde3daf002c890
I20240502 14:27:10.358235 18546 catalog_manager.cc:4830] TS c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409): tablet 8937c809610542d88fdde3daf002c890 (table rebalance_test_table_1 [id=3670a8bef9694c029d2128006cc913d3]) successfully deleted
I20240502 14:27:10.578148 19076 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:10.694968 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:10.700260 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:10.719854 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:10.725843 21007 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:10.761158 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:10.774436 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:10.799945 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:10.861408 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:10.878752 19076 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:10.916507 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:10.933115 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:10.942811 19076 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:10.979043 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:10.986146 21007 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:11.019484 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:11.024142 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:11.026201 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:11.045284 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:11.046736 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:11.047197 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:11.051519 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:11.072762 21007 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:11.118184 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:11.235983 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:11.249191 19076 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:11.277281 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:11.289767 21007 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:11.307304 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:11.311654 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:11.314095 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:11.356699 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:11.546965 19097 consensus_queue.cc:237] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:27:11.558284 20759 raft_consensus.cc:1261] T e5fb3b049ba94af998791640bd0e31c5 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:27:11.559409 25401 consensus_queue.cc:1035] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:11.567704 18701 raft_consensus.cc:1261] T e5fb3b049ba94af998791640bd0e31c5 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:27:11.570044 25401 consensus_queue.cc:1035] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:11.571246 18833 consensus_queue.cc:237] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:11.590142 25401 raft_consensus.cc:2932] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:11.596778 18701 raft_consensus.cc:2932] T e5fb3b049ba94af998791640bd0e31c5 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:11.602797 20894 raft_consensus.cc:1261] T 8b8d81c275144f898dcc69a6d98d11e0 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:11.605134 18699 raft_consensus.cc:1261] T 8b8d81c275144f898dcc69a6d98d11e0 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:11.605759 25645 consensus_queue.cc:1035] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:11.606516 25508 consensus_queue.cc:1035] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:11.609746 19076 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:11.619000 18558 catalog_manager.cc:5484] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: config changed from index 4 to 5, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:11.625545 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:11.653899 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:11.655480 18833 consensus_queue.cc:237] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:11.659871 20759 raft_consensus.cc:2932] T e5fb3b049ba94af998791640bd0e31c5 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:11.671171 21028 raft_consensus.cc:1261] T 8b8d81c275144f898dcc69a6d98d11e0 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:11.672576 25645 raft_consensus.cc:2932] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:11.676369 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:11.678782 25508 consensus_queue.cc:1035] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:11.680917 18699 raft_consensus.cc:2932] T 8b8d81c275144f898dcc69a6d98d11e0 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:11.686058 18560 catalog_manager.cc:5484] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index 5 to 6, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:11.684288 20894 raft_consensus.cc:2932] T 8b8d81c275144f898dcc69a6d98d11e0 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:11.703542 20626 raft_consensus.cc:1261] T 2c2afaffde96462eacd0d32bbdec23b1 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:11.703806 18965 raft_consensus.cc:1261] T 2c2afaffde96462eacd0d32bbdec23b1 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:11.704530 25508 consensus_queue.cc:1035] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:11.704994 21029 raft_consensus.cc:1261] T 2c2afaffde96462eacd0d32bbdec23b1 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:11.705111 25508 consensus_queue.cc:1035] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:11.705639 25508 consensus_queue.cc:1035] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:11.743077 25508 raft_consensus.cc:2932] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:11.745930 20626 raft_consensus.cc:2932] T 2c2afaffde96462eacd0d32bbdec23b1 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:11.749346 21029 raft_consensus.cc:2932] T 2c2afaffde96462eacd0d32bbdec23b1 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:11.754526 18552 catalog_manager.cc:5484] T 2c2afaffde96462eacd0d32bbdec23b1 P a834edffbcb2469aae598b069aab0681 reported cstate change: config changed from index 5 to 6, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:11.782248 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:11.783756 18681 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c2afaffde96462eacd0d32bbdec23b1 with delete_type TABLET_DATA_TOMBSTONED (TS 2db304b538e14857b7b1cc272d0db94b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54454
I20240502 14:27:11.783990 18812 tablet_service.cc:1514] Processing DeleteTablet for tablet e5fb3b049ba94af998791640bd0e31c5 with delete_type TABLET_DATA_TOMBSTONED (TS fec04c50dc5b49878ffa3c3873df32e4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:60970
I20240502 14:27:11.804884 25675 tablet_replica.cc:330] T 2c2afaffde96462eacd0d32bbdec23b1 P 2db304b538e14857b7b1cc272d0db94b: stopping tablet replica
I20240502 14:27:11.805251 25675 raft_consensus.cc:2229] T 2c2afaffde96462eacd0d32bbdec23b1 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:11.805282 25676 tablet_replica.cc:330] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4: stopping tablet replica
I20240502 14:27:11.805514 25676 raft_consensus.cc:2229] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:11.805550 25675 raft_consensus.cc:2258] T 2c2afaffde96462eacd0d32bbdec23b1 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:11.805851 25676 raft_consensus.cc:2258] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:11.806304 25675 ts_tablet_manager.cc:1875] T 2c2afaffde96462eacd0d32bbdec23b1 P 2db304b538e14857b7b1cc272d0db94b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:11.810186 18945 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b8d81c275144f898dcc69a6d98d11e0 with delete_type TABLET_DATA_TOMBSTONED (TS c902805060df4df9b5599d0e94b2dc06 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55752
I20240502 14:27:11.812610 25676 ts_tablet_manager.cc:1875] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:11.813297 25675 ts_tablet_manager.cc:1888] T 2c2afaffde96462eacd0d32bbdec23b1 P 2db304b538e14857b7b1cc272d0db94b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:27:11.813411 25675 log.cc:1196] T 2c2afaffde96462eacd0d32bbdec23b1 P 2db304b538e14857b7b1cc272d0db94b: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal/wals/2c2afaffde96462eacd0d32bbdec23b1
I20240502 14:27:11.814396 18544 catalog_manager.cc:4830] TS 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749): tablet 2c2afaffde96462eacd0d32bbdec23b1 (table rebalance_test_table_9 [id=a9acd50a500c4f84a41c8d9ba23ae431]) successfully deleted
I20240502 14:27:11.823815 25680 tablet_replica.cc:330] T 8b8d81c275144f898dcc69a6d98d11e0 P c902805060df4df9b5599d0e94b2dc06: stopping tablet replica
I20240502 14:27:11.824049 25680 raft_consensus.cc:2229] T 8b8d81c275144f898dcc69a6d98d11e0 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:11.824410 25680 raft_consensus.cc:2258] T 8b8d81c275144f898dcc69a6d98d11e0 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:11.825233 25680 ts_tablet_manager.cc:1875] T 8b8d81c275144f898dcc69a6d98d11e0 P c902805060df4df9b5599d0e94b2dc06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:11.828002 21028 raft_consensus.cc:2932] T 8b8d81c275144f898dcc69a6d98d11e0 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:11.832962 25680 ts_tablet_manager.cc:1888] T 8b8d81c275144f898dcc69a6d98d11e0 P c902805060df4df9b5599d0e94b2dc06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:27:11.833086 25680 log.cc:1196] T 8b8d81c275144f898dcc69a6d98d11e0 P c902805060df4df9b5599d0e94b2dc06: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal/wals/8b8d81c275144f898dcc69a6d98d11e0
I20240502 14:27:11.834638 18965 raft_consensus.cc:2932] T 2c2afaffde96462eacd0d32bbdec23b1 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:11.835270 25676 ts_tablet_manager.cc:1888] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:27:11.835604 25676 log.cc:1196] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/wals/e5fb3b049ba94af998791640bd0e31c5
I20240502 14:27:11.836283 18543 catalog_manager.cc:4830] TS fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009): tablet e5fb3b049ba94af998791640bd0e31c5 (table rebalance_test_table_6 [id=995bc008b9f94332bfa2c42a77e41bb2]) successfully deleted
I20240502 14:27:11.836870 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:11.840094 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:11.840160 18546 catalog_manager.cc:4830] TS c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409): tablet 8b8d81c275144f898dcc69a6d98d11e0 (table rebalance_test_table_7 [id=b260896c8b8744c0a1b4f691850aec93]) successfully deleted
I20240502 14:27:11.886862 21007 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:12.117663 18833 consensus_queue.cc:237] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:27:12.167537 20626 raft_consensus.cc:1261] T 9580f3fdf7084cc0923b1b16bfedd450 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:27:12.168344 18965 raft_consensus.cc:1261] T 9580f3fdf7084cc0923b1b16bfedd450 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:27:12.169342 20894 raft_consensus.cc:1261] T 9580f3fdf7084cc0923b1b16bfedd450 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:27:12.170004 25644 consensus_queue.cc:1035] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:12.170557 25644 consensus_queue.cc:1035] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:12.170969 25644 consensus_queue.cc:1035] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:12.177887 21028 raft_consensus.cc:1261] T 9580f3fdf7084cc0923b1b16bfedd450 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:27:12.178392 25508 consensus_queue.cc:1035] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:12.249926 25508 raft_consensus.cc:2932] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:12.251710 18963 raft_consensus.cc:2932] T 9580f3fdf7084cc0923b1b16bfedd450 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:12.261945 18558 catalog_manager.cc:5484] T 9580f3fdf7084cc0923b1b16bfedd450 P c902805060df4df9b5599d0e94b2dc06 reported cstate change: config changed from index 6 to 7, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:12.266641 20627 raft_consensus.cc:2932] T 9580f3fdf7084cc0923b1b16bfedd450 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:12.267737 20894 raft_consensus.cc:2932] T 9580f3fdf7084cc0923b1b16bfedd450 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:12.277130 18681 tablet_service.cc:1514] Processing DeleteTablet for tablet 9580f3fdf7084cc0923b1b16bfedd450 with delete_type TABLET_DATA_TOMBSTONED (TS 2db304b538e14857b7b1cc272d0db94b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54454
I20240502 14:27:12.277563 25675 tablet_replica.cc:330] T 9580f3fdf7084cc0923b1b16bfedd450 P 2db304b538e14857b7b1cc272d0db94b: stopping tablet replica
I20240502 14:27:12.277757 25675 raft_consensus.cc:2229] T 9580f3fdf7084cc0923b1b16bfedd450 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:12.277168 21029 raft_consensus.cc:2932] T 9580f3fdf7084cc0923b1b16bfedd450 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:12.278097 25675 raft_consensus.cc:2258] T 9580f3fdf7084cc0923b1b16bfedd450 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:12.278828 25675 ts_tablet_manager.cc:1875] T 9580f3fdf7084cc0923b1b16bfedd450 P 2db304b538e14857b7b1cc272d0db94b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:12.287796 25675 ts_tablet_manager.cc:1888] T 9580f3fdf7084cc0923b1b16bfedd450 P 2db304b538e14857b7b1cc272d0db94b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:27:12.287990 25675 log.cc:1196] T 9580f3fdf7084cc0923b1b16bfedd450 P 2db304b538e14857b7b1cc272d0db94b: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal/wals/9580f3fdf7084cc0923b1b16bfedd450
I20240502 14:27:12.288811 18544 catalog_manager.cc:4830] TS 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749): tablet 9580f3fdf7084cc0923b1b16bfedd450 (table rebalance_test_table_0 [id=27066b01c29c4a1380ee4753a1c036d3]) successfully deleted
I20240502 14:27:12.292300 19076 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:12.328439 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:12.369624 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:12.369794 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:12.388326 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:12.400100 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:12.441417 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:12.463099 21007 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:12.615402 19076 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:12.624634 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:12.652941 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:12.675544 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:12.719779 21028 consensus_queue.cc:237] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:12.728675 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:12.729768 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:12.770535 19097 raft_consensus.cc:1261] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [term 2 FOLLOWER]: Refusing update from remote peer ff6acba046404ef0a6f41ad5e09d852f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:27:12.771163 24273 consensus_queue.cc:1035] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:12.786470 18699 raft_consensus.cc:1261] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Refusing update from remote peer ff6acba046404ef0a6f41ad5e09d852f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:27:12.787055 24273 consensus_queue.cc:1035] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:12.797171 21007 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:12.798174 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:12.801292 25850 raft_consensus.cc:2932] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:12.801960 19097 raft_consensus.cc:2932] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:12.808066 18558 catalog_manager.cc:5484] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: config changed from index 4 to 6, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New cstate: current_term: 2 leader_uuid: "ff6acba046404ef0a6f41ad5e09d852f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:12.829942 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:12.847240 18699 raft_consensus.cc:2932] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:12.860266 18945 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ecc6539b85d40e49df7ffc5f93cb801 with delete_type TABLET_DATA_TOMBSTONED (TS c902805060df4df9b5599d0e94b2dc06 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55752
I20240502 14:27:12.869887 25857 tablet_replica.cc:330] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06: stopping tablet replica
I20240502 14:27:12.870085 25857 raft_consensus.cc:2229] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:12.870221 25857 raft_consensus.cc:2258] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:12.887602 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:12.897154 25857 ts_tablet_manager.cc:1875] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:12.912786 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:12.914608 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:12.918627 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:12.928730 25857 ts_tablet_manager.cc:1888] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 14:27:12.953894 21007 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:12.954455 19076 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:12.959164 25857 log.cc:1196] T 1ecc6539b85d40e49df7ffc5f93cb801 P c902805060df4df9b5599d0e94b2dc06: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal/wals/1ecc6539b85d40e49df7ffc5f93cb801
I20240502 14:27:12.963698 18546 catalog_manager.cc:4830] TS c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409): tablet 1ecc6539b85d40e49df7ffc5f93cb801 (table rebalance_test_table_9 [id=a9acd50a500c4f84a41c8d9ba23ae431]) successfully deleted
I20240502 14:27:12.993399 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:13.175913 19093 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "332e4b23751b422ab7bd17b80c93701d"
dest_uuid: "007b2e79dc424a6f80141180f072f2d8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37270
I20240502 14:27:13.176106 19093 raft_consensus.cc:595] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:27:13.241572 19093 consensus_queue.cc:237] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:27:13.256175 20894 raft_consensus.cc:1261] T 8937c809610542d88fdde3daf002c890 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:27:13.256958 20627 raft_consensus.cc:1261] T 8937c809610542d88fdde3daf002c890 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:27:13.258174 25401 consensus_queue.cc:1035] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:27:13.258862 25401 consensus_queue.cc:1035] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:27:13.266748 25401 raft_consensus.cc:979] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8: : Instructing follower ff6acba046404ef0a6f41ad5e09d852f to start an election
I20240502 14:27:13.266858 25401 raft_consensus.cc:1067] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Signalling peer ff6acba046404ef0a6f41ad5e09d852f to start an election
I20240502 14:27:13.267426 21028 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "332e4b23751b422ab7bd17b80c93701d"
dest_uuid: "ff6acba046404ef0a6f41ad5e09d852f"
from {username='slave'} at 127.5.107.68:40869
I20240502 14:27:13.267686 21028 raft_consensus.cc:482] T 332e4b23751b422ab7bd17b80c93701d P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:27:13.267827 21028 raft_consensus.cc:3037] T 332e4b23751b422ab7bd17b80c93701d P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:27:13.273973 21028 raft_consensus.cc:504] T 332e4b23751b422ab7bd17b80c93701d P ff6acba046404ef0a6f41ad5e09d852f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:13.277029 21028 leader_election.cc:288] T 332e4b23751b422ab7bd17b80c93701d P ff6acba046404ef0a6f41ad5e09d852f [CANDIDATE]: Term 2 election: Requested vote from peers 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463), c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409), fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
I20240502 14:27:13.277099 19097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "332e4b23751b422ab7bd17b80c93701d" candidate_uuid: "ff6acba046404ef0a6f41ad5e09d852f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "007b2e79dc424a6f80141180f072f2d8"
I20240502 14:27:13.277292 19097 raft_consensus.cc:3032] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:27:13.277364 19097 raft_consensus.cc:729] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Running, Role: LEADER
I20240502 14:27:13.279415 20626 raft_consensus.cc:2932] T 8937c809610542d88fdde3daf002c890 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:13.280745 18963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "332e4b23751b422ab7bd17b80c93701d" candidate_uuid: "ff6acba046404ef0a6f41ad5e09d852f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c902805060df4df9b5599d0e94b2dc06"
I20240502 14:27:13.281013 18963 raft_consensus.cc:3037] T 332e4b23751b422ab7bd17b80c93701d P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:27:13.281383 18833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "332e4b23751b422ab7bd17b80c93701d" candidate_uuid: "ff6acba046404ef0a6f41ad5e09d852f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "fec04c50dc5b49878ffa3c3873df32e4"
I20240502 14:27:13.281622 18833 raft_consensus.cc:3037] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:27:13.278546 25401 raft_consensus.cc:2932] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:13.287823 18553 catalog_manager.cc:5484] T 8937c809610542d88fdde3daf002c890 P a834edffbcb2469aae598b069aab0681 reported cstate change: config changed from index 7 to 8, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:13.291908 18812 tablet_service.cc:1514] Processing DeleteTablet for tablet 8937c809610542d88fdde3daf002c890 with delete_type TABLET_DATA_TOMBSTONED (TS fec04c50dc5b49878ffa3c3873df32e4 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:60970
I20240502 14:27:13.284905 18833 raft_consensus.cc:2454] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff6acba046404ef0a6f41ad5e09d852f in term 2.
I20240502 14:27:13.298931 18963 raft_consensus.cc:2454] T 332e4b23751b422ab7bd17b80c93701d P c902805060df4df9b5599d0e94b2dc06 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff6acba046404ef0a6f41ad5e09d852f in term 2.
I20240502 14:27:13.300559 20964 leader_election.cc:302] T 332e4b23751b422ab7bd17b80c93701d P ff6acba046404ef0a6f41ad5e09d852f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: c902805060df4df9b5599d0e94b2dc06, fec04c50dc5b49878ffa3c3873df32e4, ff6acba046404ef0a6f41ad5e09d852f; no voters:
I20240502 14:27:13.277604 19097 consensus_queue.cc:260] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:13.300969 19097 raft_consensus.cc:3037] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:27:13.301635 25853 raft_consensus.cc:2781] T 332e4b23751b422ab7bd17b80c93701d P ff6acba046404ef0a6f41ad5e09d852f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:27:13.301753 25853 raft_consensus.cc:686] T 332e4b23751b422ab7bd17b80c93701d P ff6acba046404ef0a6f41ad5e09d852f [term 2 LEADER]: Becoming Leader. State: Replica: ff6acba046404ef0a6f41ad5e09d852f, State: Running, Role: LEADER
I20240502 14:27:13.302000 25853 consensus_queue.cc:237] T 332e4b23751b422ab7bd17b80c93701d P ff6acba046404ef0a6f41ad5e09d852f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:13.310196 19097 raft_consensus.cc:2454] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff6acba046404ef0a6f41ad5e09d852f in term 2.
I20240502 14:27:13.346343 18553 catalog_manager.cc:5484] T 332e4b23751b422ab7bd17b80c93701d P ff6acba046404ef0a6f41ad5e09d852f reported cstate change: term changed from 1 to 2, leader changed from 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) to ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72). New cstate: current_term: 2 leader_uuid: "ff6acba046404ef0a6f41ad5e09d852f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:13.351032 20894 raft_consensus.cc:2932] T 8937c809610542d88fdde3daf002c890 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:13.360697 25919 tablet_replica.cc:330] T 8937c809610542d88fdde3daf002c890 P fec04c50dc5b49878ffa3c3873df32e4: stopping tablet replica
I20240502 14:27:13.360962 25919 raft_consensus.cc:2229] T 8937c809610542d88fdde3daf002c890 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:13.362016 25919 raft_consensus.cc:2258] T 8937c809610542d88fdde3daf002c890 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:13.368459 25919 ts_tablet_manager.cc:1875] T 8937c809610542d88fdde3daf002c890 P fec04c50dc5b49878ffa3c3873df32e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:13.374140 21028 raft_consensus.cc:1261] T 8937c809610542d88fdde3daf002c890 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:27:13.378702 25919 ts_tablet_manager.cc:1888] T 8937c809610542d88fdde3daf002c890 P fec04c50dc5b49878ffa3c3873df32e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 14:27:13.378859 25919 log.cc:1196] T 8937c809610542d88fdde3daf002c890 P fec04c50dc5b49878ffa3c3873df32e4: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/wals/8937c809610542d88fdde3daf002c890
I20240502 14:27:13.380918 18543 catalog_manager.cc:4830] TS fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009): tablet 8937c809610542d88fdde3daf002c890 (table rebalance_test_table_1 [id=3670a8bef9694c029d2128006cc913d3]) successfully deleted
I20240502 14:27:13.388083 25921 consensus_queue.cc:1035] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:27:13.402331 21028 raft_consensus.cc:2932] T 8937c809610542d88fdde3daf002c890 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:13.549141 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:13.563611 19076 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:13.588606 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:13.595538 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:13.612892 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:13.628311 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:13.647267 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:13.699321 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:13.839658 19076 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:13.863757 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:13.865293 18963 raft_consensus.cc:1261] T 332e4b23751b422ab7bd17b80c93701d P c902805060df4df9b5599d0e94b2dc06 [term 2 FOLLOWER]: Refusing update from remote peer ff6acba046404ef0a6f41ad5e09d852f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:27:13.866588 25853 consensus_queue.cc:1035] T 332e4b23751b422ab7bd17b80c93701d P ff6acba046404ef0a6f41ad5e09d852f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:13.894966 19097 raft_consensus.cc:1261] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [term 2 FOLLOWER]: Refusing update from remote peer ff6acba046404ef0a6f41ad5e09d852f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:27:13.899489 25853 consensus_queue.cc:1035] T 332e4b23751b422ab7bd17b80c93701d P ff6acba046404ef0a6f41ad5e09d852f [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:13.900830 18833 raft_consensus.cc:1261] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 [term 2 FOLLOWER]: Refusing update from remote peer ff6acba046404ef0a6f41ad5e09d852f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:27:13.912139 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:13.912683 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:13.928894 26037 consensus_queue.cc:1035] T 332e4b23751b422ab7bd17b80c93701d P ff6acba046404ef0a6f41ad5e09d852f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:13.967612 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:13.999336 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:14.014158 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:14.019498 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:14.191692 18768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.72:35871, user_credentials={real_user=slave}} blocked reactor thread for 58410us
I20240502 14:27:14.250716 19076 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:14.254292 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:14.258554 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:14.259819 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:14.343760 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:14.344671 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:14.344919 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:14.347118 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:14.458954 19097 consensus_queue.cc:237] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:27:14.490140 18699 raft_consensus.cc:1261] T 097eb705c9e24740a5be61bf3354f3d8 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:27:14.491191 25921 consensus_queue.cc:1035] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:14.493896 20626 raft_consensus.cc:1261] T 097eb705c9e24740a5be61bf3354f3d8 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:27:14.494793 25921 consensus_queue.cc:1035] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:14.535441 25921 raft_consensus.cc:2932] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:14.541719 18701 raft_consensus.cc:2932] T 097eb705c9e24740a5be61bf3354f3d8 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:14.561559 18553 catalog_manager.cc:5484] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: config changed from index 4 to 5, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:14.587852 20627 raft_consensus.cc:2932] T 097eb705c9e24740a5be61bf3354f3d8 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:14.638820 18812 tablet_service.cc:1514] Processing DeleteTablet for tablet 097eb705c9e24740a5be61bf3354f3d8 with delete_type TABLET_DATA_TOMBSTONED (TS fec04c50dc5b49878ffa3c3873df32e4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:60970
I20240502 14:27:14.639420 26153 tablet_replica.cc:330] T 097eb705c9e24740a5be61bf3354f3d8 P fec04c50dc5b49878ffa3c3873df32e4: stopping tablet replica
I20240502 14:27:14.639618 26153 raft_consensus.cc:2229] T 097eb705c9e24740a5be61bf3354f3d8 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:14.639834 26153 raft_consensus.cc:2258] T 097eb705c9e24740a5be61bf3354f3d8 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:14.641834 26153 ts_tablet_manager.cc:1875] T 097eb705c9e24740a5be61bf3354f3d8 P fec04c50dc5b49878ffa3c3873df32e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:14.652518 26153 ts_tablet_manager.cc:1888] T 097eb705c9e24740a5be61bf3354f3d8 P fec04c50dc5b49878ffa3c3873df32e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:27:14.652627 26153 log.cc:1196] T 097eb705c9e24740a5be61bf3354f3d8 P fec04c50dc5b49878ffa3c3873df32e4: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/wals/097eb705c9e24740a5be61bf3354f3d8
I20240502 14:27:14.653316 18543 catalog_manager.cc:4830] TS fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009): tablet 097eb705c9e24740a5be61bf3354f3d8 (table rebalance_test_table_4 [id=4dd18a179f134d1babc05959227a25cc]) successfully deleted
I20240502 14:27:14.710600 19076 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:14.749671 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:14.785723 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:14.816838 18699 consensus_queue.cc:237] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:27:14.833060 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:14.852300 21028 consensus_queue.cc:237] T 332e4b23751b422ab7bd17b80c93701d P ff6acba046404ef0a6f41ad5e09d852f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:14.854909 18963 raft_consensus.cc:1261] T 332e4b23751b422ab7bd17b80c93701d P c902805060df4df9b5599d0e94b2dc06 [term 2 FOLLOWER]: Refusing update from remote peer ff6acba046404ef0a6f41ad5e09d852f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:27:14.855538 25853 consensus_queue.cc:1035] T 332e4b23751b422ab7bd17b80c93701d P ff6acba046404ef0a6f41ad5e09d852f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:14.856626 18833 raft_consensus.cc:1261] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 [term 2 FOLLOWER]: Refusing update from remote peer ff6acba046404ef0a6f41ad5e09d852f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:27:14.857232 25853 consensus_queue.cc:1035] T 332e4b23751b422ab7bd17b80c93701d P ff6acba046404ef0a6f41ad5e09d852f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:14.858806 20759 raft_consensus.cc:1261] T 0ffb6e582ef04fb38dd273c13febac58 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:14.859382 26074 consensus_queue.cc:1035] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:14.860116 25853 raft_consensus.cc:2932] T 332e4b23751b422ab7bd17b80c93701d P ff6acba046404ef0a6f41ad5e09d852f [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:14.863785 21029 raft_consensus.cc:1261] T 0ffb6e582ef04fb38dd273c13febac58 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:14.864502 24416 consensus_queue.cc:1035] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:14.865422 18833 raft_consensus.cc:2932] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:14.872891 18551 catalog_manager.cc:5484] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index 4 to 6, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New cstate: current_term: 2 leader_uuid: "ff6acba046404ef0a6f41ad5e09d852f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:14.877061 18963 raft_consensus.cc:1261] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:14.878270 19076 tablet_service.cc:1514] Processing DeleteTablet for tablet 332e4b23751b422ab7bd17b80c93701d with delete_type TABLET_DATA_TOMBSTONED (TS 007b2e79dc424a6f80141180f072f2d8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41468
I20240502 14:27:14.878667 26074 consensus_queue.cc:1035] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:14.906502 18963 raft_consensus.cc:2932] T 332e4b23751b422ab7bd17b80c93701d P c902805060df4df9b5599d0e94b2dc06 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:14.916108 26074 raft_consensus.cc:2932] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:14.918016 20760 raft_consensus.cc:2932] T 0ffb6e582ef04fb38dd273c13febac58 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:14.935214 18551 catalog_manager.cc:5484] T 0ffb6e582ef04fb38dd273c13febac58 P a14c220ad2cc4bf9bd7dad65f4803eb0 reported cstate change: config changed from index 5 to 6, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:14.937700 19077 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:14.939582 18812 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ffb6e582ef04fb38dd273c13febac58 with delete_type TABLET_DATA_TOMBSTONED (TS fec04c50dc5b49878ffa3c3873df32e4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60970
I20240502 14:27:14.939893 26153 tablet_replica.cc:330] T 0ffb6e582ef04fb38dd273c13febac58 P fec04c50dc5b49878ffa3c3873df32e4: stopping tablet replica
I20240502 14:27:14.940065 26153 raft_consensus.cc:2229] T 0ffb6e582ef04fb38dd273c13febac58 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:14.940237 26153 raft_consensus.cc:2258] T 0ffb6e582ef04fb38dd273c13febac58 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:14.940938 26153 ts_tablet_manager.cc:1875] T 0ffb6e582ef04fb38dd273c13febac58 P fec04c50dc5b49878ffa3c3873df32e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:14.946280 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:14.949206 26153 ts_tablet_manager.cc:1888] T 0ffb6e582ef04fb38dd273c13febac58 P fec04c50dc5b49878ffa3c3873df32e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:27:14.949074 18963 raft_consensus.cc:2932] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:14.954381 26153 log.cc:1196] T 0ffb6e582ef04fb38dd273c13febac58 P fec04c50dc5b49878ffa3c3873df32e4: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/wals/0ffb6e582ef04fb38dd273c13febac58
I20240502 14:27:14.955396 18543 catalog_manager.cc:4830] TS fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009): tablet 0ffb6e582ef04fb38dd273c13febac58 (table rebalance_test_table_1 [id=3670a8bef9694c029d2128006cc913d3]) successfully deleted
I20240502 14:27:14.984854 26206 tablet_replica.cc:330] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8: stopping tablet replica
I20240502 14:27:14.985139 26206 raft_consensus.cc:2229] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:14.985430 26206 raft_consensus.cc:2258] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:14.988106 21028 raft_consensus.cc:2932] T 0ffb6e582ef04fb38dd273c13febac58 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:15.002243 26206 ts_tablet_manager.cc:1875] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:15.046770 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:15.063241 26206 ts_tablet_manager.cc:1888] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 14:27:15.063421 26206 log.cc:1196] T 332e4b23751b422ab7bd17b80c93701d P 007b2e79dc424a6f80141180f072f2d8: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal/wals/332e4b23751b422ab7bd17b80c93701d
I20240502 14:27:15.064301 18546 catalog_manager.cc:4830] TS 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463): tablet 332e4b23751b422ab7bd17b80c93701d (table rebalance_test_table_3 [id=210510c054044948932a30db816646aa]) successfully deleted
I20240502 14:27:15.093189 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:15.121330 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:15.134418 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:15.181097 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:15.202736 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:15.203310 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:15.208741 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:15.210000 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:15.273116 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:15.538414 18833 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "522014370f0a4cc3bd31aa00d6a05c5e"
dest_uuid: "fec04c50dc5b49878ffa3c3873df32e4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51864
I20240502 14:27:15.538738 18833 raft_consensus.cc:595] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:27:15.605893 19076 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:15.633092 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:15.674669 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:27:15.681586 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:15.705499 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:15.705852 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:15.709364 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:15.712620 26072 raft_consensus.cc:979] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4: : Instructing follower 2db304b538e14857b7b1cc272d0db94b to start an election
I20240502 14:27:15.712740 26072 raft_consensus.cc:1067] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Signalling peer 2db304b538e14857b7b1cc272d0db94b to start an election
I20240502 14:27:15.713603 18699 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "522014370f0a4cc3bd31aa00d6a05c5e"
dest_uuid: "2db304b538e14857b7b1cc272d0db94b"
from {username='slave'} at 127.5.107.66:57219
I20240502 14:27:15.713805 18699 raft_consensus.cc:482] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:27:15.713892 18699 raft_consensus.cc:3037] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:27:15.716718 18699 raft_consensus.cc:504] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:27:15.718573 18833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "522014370f0a4cc3bd31aa00d6a05c5e" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "fec04c50dc5b49878ffa3c3873df32e4"
I20240502 14:27:15.718745 18833 raft_consensus.cc:3032] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:27:15.718814 18833 raft_consensus.cc:729] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Running, Role: LEADER
I20240502 14:27:15.719056 18833 consensus_queue.cc:260] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:27:15.719520 18833 raft_consensus.cc:3037] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:27:15.722510 18833 raft_consensus.cc:2454] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2db304b538e14857b7b1cc272d0db94b in term 2.
I20240502 14:27:15.723790 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "522014370f0a4cc3bd31aa00d6a05c5e" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ff6acba046404ef0a6f41ad5e09d852f"
I20240502 14:27:15.724004 21029 raft_consensus.cc:3037] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:27:15.726830 21029 raft_consensus.cc:2454] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2db304b538e14857b7b1cc272d0db94b in term 2.
I20240502 14:27:15.727243 18636 leader_election.cc:302] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b, fec04c50dc5b49878ffa3c3873df32e4, ff6acba046404ef0a6f41ad5e09d852f; no voters:
I20240502 14:27:15.727514 26074 raft_consensus.cc:2781] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:27:15.727613 26074 raft_consensus.cc:686] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 2 LEADER]: Becoming Leader. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Running, Role: LEADER
I20240502 14:27:15.727864 26074 consensus_queue.cc:237] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:27:15.733934 20627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "522014370f0a4cc3bd31aa00d6a05c5e" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "a834edffbcb2469aae598b069aab0681"
I20240502 14:27:15.734225 20627 raft_consensus.cc:3037] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:27:15.736855 20627 raft_consensus.cc:2454] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2db304b538e14857b7b1cc272d0db94b in term 2.
I20240502 14:27:15.737514 18699 leader_election.cc:288] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 2 election: Requested vote from peers fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009), 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463), ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119)
I20240502 14:27:15.752069 19093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "522014370f0a4cc3bd31aa00d6a05c5e" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "007b2e79dc424a6f80141180f072f2d8"
I20240502 14:27:15.752267 19093 raft_consensus.cc:3037] T 522014370f0a4cc3bd31aa00d6a05c5e P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:27:15.754951 19093 raft_consensus.cc:2454] T 522014370f0a4cc3bd31aa00d6a05c5e P 007b2e79dc424a6f80141180f072f2d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2db304b538e14857b7b1cc272d0db94b in term 2.
I20240502 14:27:15.760967 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:15.767019 18560 catalog_manager.cc:5484] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b reported cstate change: term changed from 1 to 2, leader changed from fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) to 2db304b538e14857b7b1cc272d0db94b (127.5.107.65). New cstate: current_term: 2 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:15.983345 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:16.000104 19076 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:16.005342 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:16.035468 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:16.050268 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:16.060474 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:16.098093 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:16.116688 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:16.137629 20627 raft_consensus.cc:1261] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [term 2 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:27:16.138482 26074 consensus_queue.cc:1035] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:16.175199 18833 raft_consensus.cc:1261] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [term 2 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:27:16.175949 26074 consensus_queue.cc:1035] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:16.210983 21029 raft_consensus.cc:1261] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f [term 2 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:27:16.211951 26074 consensus_queue.cc:1035] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:16.230434 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:16.243850 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:16.264067 19093 raft_consensus.cc:1261] T 522014370f0a4cc3bd31aa00d6a05c5e P 007b2e79dc424a6f80141180f072f2d8 [term 2 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:27:16.265343 26074 consensus_queue.cc:1035] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20240502 14:27:16.268687 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:16.307348 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:16.350283 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:16.366660 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:16.373162 19076 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:16.425778 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:16.460724 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:16.474123 19076 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:16.476274 19097 consensus_queue.cc:237] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:27:16.502586 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:16.504374 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:16.558777 20627 raft_consensus.cc:1261] T 6a992d12aee7416096d1353533397386 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:16.559553 25401 consensus_queue.cc:1035] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:16.566681 20760 raft_consensus.cc:1261] T 6a992d12aee7416096d1353533397386 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:16.567335 25401 consensus_queue.cc:1035] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:16.577474 18699 raft_consensus.cc:1261] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:16.578548 25401 consensus_queue.cc:1035] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:16.607564 25401 raft_consensus.cc:2932] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:16.616693 20759 raft_consensus.cc:2932] T 6a992d12aee7416096d1353533397386 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:16.639040 18699 raft_consensus.cc:2932] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:16.647001 18554 catalog_manager.cc:5484] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b reported cstate change: config changed from index 5 to 6, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:16.647445 20627 raft_consensus.cc:2932] T 6a992d12aee7416096d1353533397386 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:16.664180 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:16.698453 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:16.755266 18945 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a992d12aee7416096d1353533397386 with delete_type TABLET_DATA_TOMBSTONED (TS c902805060df4df9b5599d0e94b2dc06 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55752
I20240502 14:27:16.764238 19076 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:16.765085 26481 tablet_replica.cc:330] T 6a992d12aee7416096d1353533397386 P c902805060df4df9b5599d0e94b2dc06: stopping tablet replica
I20240502 14:27:16.795722 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:16.803282 26481 raft_consensus.cc:2229] T 6a992d12aee7416096d1353533397386 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:16.803458 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:16.803629 26481 raft_consensus.cc:2258] T 6a992d12aee7416096d1353533397386 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:16.804419 26481 ts_tablet_manager.cc:1875] T 6a992d12aee7416096d1353533397386 P c902805060df4df9b5599d0e94b2dc06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:16.851140 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:16.857784 26481 ts_tablet_manager.cc:1888] T 6a992d12aee7416096d1353533397386 P c902805060df4df9b5599d0e94b2dc06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:27:16.858085 26481 log.cc:1196] T 6a992d12aee7416096d1353533397386 P c902805060df4df9b5599d0e94b2dc06: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal/wals/6a992d12aee7416096d1353533397386
I20240502 14:27:16.859324 18546 catalog_manager.cc:4830] TS c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409): tablet 6a992d12aee7416096d1353533397386 (table rebalance_test_table_5 [id=ff5cff7e35c746a5bc64cc06f18bf7d9]) successfully deleted
I20240502 14:27:16.866957 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:16.930060 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:16.950836 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:16.952620 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:16.969338 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:17.045576 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:17.186395 18963 consensus_queue.cc:237] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:27:17.218961 20892 raft_consensus.cc:1261] T cf2a6426ae6c438f942d39530da6e3e3 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:17.219686 26262 consensus_queue.cc:1035] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:17.266571 19097 consensus_queue.cc:237] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:27:17.269568 20627 raft_consensus.cc:1261] T 6a992d12aee7416096d1353533397386 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:27:17.270097 20759 raft_consensus.cc:1261] T 6a992d12aee7416096d1353533397386 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:27:17.270159 25401 consensus_queue.cc:1035] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:17.270807 26455 consensus_queue.cc:1035] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:17.272697 26539 raft_consensus.cc:2932] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:17.275071 25401 raft_consensus.cc:2932] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:17.278424 20759 raft_consensus.cc:2932] T 6a992d12aee7416096d1353533397386 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:17.278657 20892 raft_consensus.cc:2932] T cf2a6426ae6c438f942d39530da6e3e3 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:17.282614 20627 raft_consensus.cc:2932] T 6a992d12aee7416096d1353533397386 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:17.286327 18560 catalog_manager.cc:5484] T cf2a6426ae6c438f942d39530da6e3e3 P 7ab6d395b30d4afb9821b61c9454430b reported cstate change: config changed from index 5 to 6, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:17.303839 18551 catalog_manager.cc:5484] T 6a992d12aee7416096d1353533397386 P a834edffbcb2469aae598b069aab0681 reported cstate change: config changed from index 6 to 7, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:17.339556 18699 consensus_queue.cc:237] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:27:17.342978 18681 tablet_service.cc:1514] Processing DeleteTablet for tablet cf2a6426ae6c438f942d39530da6e3e3 with delete_type TABLET_DATA_TOMBSTONED (TS 2db304b538e14857b7b1cc272d0db94b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54454
I20240502 14:27:17.344264 18680 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a992d12aee7416096d1353533397386 with delete_type TABLET_DATA_TOMBSTONED (TS 2db304b538e14857b7b1cc272d0db94b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54454
I20240502 14:27:17.348650 18963 consensus_queue.cc:237] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:27:17.352154 20892 raft_consensus.cc:1261] T cf2a6426ae6c438f942d39530da6e3e3 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:27:17.355118 26544 consensus_queue.cc:1035] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:27:17.356567 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf2a6426ae6c438f942d39530da6e3e3. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:27:17.358930 26576 tablet_replica.cc:330] T cf2a6426ae6c438f942d39530da6e3e3 P 2db304b538e14857b7b1cc272d0db94b: stopping tablet replica
I20240502 14:27:17.359140 26576 raft_consensus.cc:2229] T cf2a6426ae6c438f942d39530da6e3e3 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:17.359499 26576 raft_consensus.cc:2258] T cf2a6426ae6c438f942d39530da6e3e3 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:17.360271 26576 ts_tablet_manager.cc:1875] T cf2a6426ae6c438f942d39530da6e3e3 P 2db304b538e14857b7b1cc272d0db94b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:17.361718 26544 raft_consensus.cc:2932] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:17.367221 20892 raft_consensus.cc:2932] T cf2a6426ae6c438f942d39530da6e3e3 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:17.369293 26576 ts_tablet_manager.cc:1888] T cf2a6426ae6c438f942d39530da6e3e3 P 2db304b538e14857b7b1cc272d0db94b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:27:17.369488 26576 log.cc:1196] T cf2a6426ae6c438f942d39530da6e3e3 P 2db304b538e14857b7b1cc272d0db94b: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal/wals/cf2a6426ae6c438f942d39530da6e3e3
I20240502 14:27:17.370404 26576 tablet_replica.cc:330] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b: stopping tablet replica
I20240502 14:27:17.370648 26576 raft_consensus.cc:2229] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:17.371069 26576 raft_consensus.cc:2258] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:17.371104 18544 catalog_manager.cc:4830] TS 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749): tablet cf2a6426ae6c438f942d39530da6e3e3 (table rebalance_test_table_0 [id=27066b01c29c4a1380ee4753a1c036d3]) successfully deleted
I20240502 14:27:17.377643 26576 ts_tablet_manager.cc:1875] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:17.378593 18546 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet cf2a6426ae6c438f942d39530da6e3e3 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:27:17.387580 18560 catalog_manager.cc:5484] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 reported cstate change: config changed from index 6 to 7, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:17.391595 20759 raft_consensus.cc:1261] T 0ffb6e582ef04fb38dd273c13febac58 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:27:17.392181 24416 consensus_queue.cc:1035] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:17.396167 26576 ts_tablet_manager.cc:1888] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:27:17.396570 21029 raft_consensus.cc:1261] T 0ffb6e582ef04fb38dd273c13febac58 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:27:17.396872 26576 log.cc:1196] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal/wals/6a992d12aee7416096d1353533397386
I20240502 14:27:17.397233 24416 consensus_queue.cc:1035] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:17.398308 18544 catalog_manager.cc:4830] TS 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749): tablet 6a992d12aee7416096d1353533397386 (table rebalance_test_table_5 [id=ff5cff7e35c746a5bc64cc06f18bf7d9]) successfully deleted
I20240502 14:27:17.419202 20759 raft_consensus.cc:2932] T 0ffb6e582ef04fb38dd273c13febac58 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:17.427423 21028 raft_consensus.cc:2932] T 0ffb6e582ef04fb38dd273c13febac58 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:17.437450 18560 catalog_manager.cc:5484] T 0ffb6e582ef04fb38dd273c13febac58 P a14c220ad2cc4bf9bd7dad65f4803eb0 reported cstate change: config changed from index 6 to 7, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:17.443145 18945 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ffb6e582ef04fb38dd273c13febac58 with delete_type TABLET_DATA_TOMBSTONED (TS c902805060df4df9b5599d0e94b2dc06 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55752
I20240502 14:27:17.444312 26594 tablet_replica.cc:330] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06: stopping tablet replica
I20240502 14:27:17.446273 26584 raft_consensus.cc:2932] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:17.449478 26594 raft_consensus.cc:2229] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:17.449896 26594 raft_consensus.cc:2258] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:17.450872 26594 ts_tablet_manager.cc:1875] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:17.487955 26594 ts_tablet_manager.cc:1888] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:27:17.488083 26594 log.cc:1196] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal/wals/0ffb6e582ef04fb38dd273c13febac58
I20240502 14:27:17.491068 18546 catalog_manager.cc:4830] TS c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409): tablet 0ffb6e582ef04fb38dd273c13febac58 (table rebalance_test_table_1 [id=3670a8bef9694c029d2128006cc913d3]) successfully deleted
I20240502 14:27:17.597002 18699 consensus_queue.cc:237] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:27:17.601703 21028 raft_consensus.cc:1261] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f [term 2 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:27:17.602089 19097 raft_consensus.cc:1261] T 522014370f0a4cc3bd31aa00d6a05c5e P 007b2e79dc424a6f80141180f072f2d8 [term 2 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:27:17.602397 24416 consensus_queue.cc:1035] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:17.602756 26584 consensus_queue.cc:1035] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:17.601162 20627 raft_consensus.cc:1261] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [term 2 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:27:17.610723 24416 consensus_queue.cc:1035] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:17.629251 26584 raft_consensus.cc:2932] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:17.638021 19097 raft_consensus.cc:2932] T 522014370f0a4cc3bd31aa00d6a05c5e P 007b2e79dc424a6f80141180f072f2d8 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:17.638672 21028 raft_consensus.cc:2932] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:17.647845 18554 catalog_manager.cc:5484] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f reported cstate change: config changed from index 5 to 7, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New cstate: current_term: 2 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:17.658175 20627 raft_consensus.cc:2932] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:17.668484 18812 tablet_service.cc:1514] Processing DeleteTablet for tablet 522014370f0a4cc3bd31aa00d6a05c5e with delete_type TABLET_DATA_TOMBSTONED (TS fec04c50dc5b49878ffa3c3873df32e4 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60970
I20240502 14:27:17.686671 26668 tablet_replica.cc:330] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4: stopping tablet replica
I20240502 14:27:17.686939 26668 raft_consensus.cc:2229] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:17.687150 26668 raft_consensus.cc:2258] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:17.687938 26668 ts_tablet_manager.cc:1875] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:17.693759 26668 ts_tablet_manager.cc:1888] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 14:27:17.693892 26668 log.cc:1196] T 522014370f0a4cc3bd31aa00d6a05c5e P fec04c50dc5b49878ffa3c3873df32e4: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/wals/522014370f0a4cc3bd31aa00d6a05c5e
I20240502 14:27:17.694988 18543 catalog_manager.cc:4830] TS fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009): tablet 522014370f0a4cc3bd31aa00d6a05c5e (table rebalance_test_table_2 [id=0e84c82d60e64dbd9149c5f9fb7ff3c8]) successfully deleted
I20240502 14:27:17.801573 19076 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:17.803560 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:17.836377 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:17.842105 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:17.852041 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:17.866724 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:17.867566 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:17.889930 26706 ts_tablet_manager.cc:902] T cf2a6426ae6c438f942d39530da6e3e3 P a14c220ad2cc4bf9bd7dad65f4803eb0: Initiating tablet copy from peer c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:27:17.916317 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:17.945123 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:17.967790 19076 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:17.983302 26706 tablet_copy_client.cc:308] T cf2a6426ae6c438f942d39530da6e3e3 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.67:45409
I20240502 14:27:17.983966 18985 tablet_copy_service.cc:140] P c902805060df4df9b5599d0e94b2dc06: Received BeginTabletCopySession request for tablet cf2a6426ae6c438f942d39530da6e3e3 from peer a14c220ad2cc4bf9bd7dad65f4803eb0 ({username='slave'} at 127.5.107.70:56629)
I20240502 14:27:17.984099 18985 tablet_copy_service.cc:161] P c902805060df4df9b5599d0e94b2dc06: Beginning new tablet copy session on tablet cf2a6426ae6c438f942d39530da6e3e3 from peer a14c220ad2cc4bf9bd7dad65f4803eb0 at {username='slave'} at 127.5.107.70:56629: session id = a14c220ad2cc4bf9bd7dad65f4803eb0-cf2a6426ae6c438f942d39530da6e3e3
I20240502 14:27:17.986819 18985 tablet_copy_source_session.cc:205] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:27:17.993340 26706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf2a6426ae6c438f942d39530da6e3e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:27:18.000798 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:18.001588 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:18.002671 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:18.015153 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:18.018985 26706 tablet_copy_client.cc:783] T cf2a6426ae6c438f942d39530da6e3e3 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Starting download of 0 data blocks...
I20240502 14:27:18.060822 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:18.064344 26706 tablet_copy_client.cc:647] T cf2a6426ae6c438f942d39530da6e3e3 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Starting download of 1 WAL segments...
I20240502 14:27:18.074252 26706 tablet_copy_client.cc:523] T cf2a6426ae6c438f942d39530da6e3e3 P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:27:18.079424 26706 tablet_bootstrap.cc:492] T cf2a6426ae6c438f942d39530da6e3e3 P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap starting.
I20240502 14:27:18.093286 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:18.181016 26706 tablet_bootstrap.cc:492] T cf2a6426ae6c438f942d39530da6e3e3 P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:27:18.181757 26706 tablet_bootstrap.cc:492] T cf2a6426ae6c438f942d39530da6e3e3 P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap complete.
I20240502 14:27:18.182200 26706 ts_tablet_manager.cc:1367] T cf2a6426ae6c438f942d39530da6e3e3 P a14c220ad2cc4bf9bd7dad65f4803eb0: Time spent bootstrapping tablet: real 0.104s user 0.044s sys 0.008s
I20240502 14:27:18.183531 26706 raft_consensus.cc:348] T cf2a6426ae6c438f942d39530da6e3e3 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:27:18.183843 26706 raft_consensus.cc:729] T cf2a6426ae6c438f942d39530da6e3e3 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a14c220ad2cc4bf9bd7dad65f4803eb0, State: Initialized, Role: FOLLOWER
I20240502 14:27:18.184121 26706 consensus_queue.cc:260] T cf2a6426ae6c438f942d39530da6e3e3 P a14c220ad2cc4bf9bd7dad65f4803eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:27:18.276450 26706 ts_tablet_manager.cc:1398] T cf2a6426ae6c438f942d39530da6e3e3 P a14c220ad2cc4bf9bd7dad65f4803eb0: Time spent starting tablet: real 0.094s user 0.024s sys 0.037s
I20240502 14:27:18.277715 18985 tablet_copy_service.cc:342] P c902805060df4df9b5599d0e94b2dc06: Request end of tablet copy session a14c220ad2cc4bf9bd7dad65f4803eb0-cf2a6426ae6c438f942d39530da6e3e3 received from {username='slave'} at 127.5.107.70:56629
I20240502 14:27:18.277890 18985 tablet_copy_service.cc:433] P c902805060df4df9b5599d0e94b2dc06: ending tablet copy session a14c220ad2cc4bf9bd7dad65f4803eb0-cf2a6426ae6c438f942d39530da6e3e3 on tablet cf2a6426ae6c438f942d39530da6e3e3 with peer a14c220ad2cc4bf9bd7dad65f4803eb0
I20240502 14:27:18.483022 19076 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:18.491660 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:18.514048 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:18.554188 19076 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:18.599632 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:18.603560 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:18.609414 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:18.640555 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:18.641937 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:18.648762 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:18.688009 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:18.694367 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:18.697696 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:18.709435 20759 raft_consensus.cc:1203] T cf2a6426ae6c438f942d39530da6e3e3 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:27:18.746562 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:18.761253 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:18.779084 18965 consensus_queue.cc:237] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:27:18.792629 18832 consensus_queue.cc:237] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:27:18.795043 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:18.811100 20759 raft_consensus.cc:1261] T 6acde0e5509340fbb52bca18645a77b7 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:27:18.811681 26072 consensus_queue.cc:1035] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:18.829826 19097 raft_consensus.cc:1261] T 6acde0e5509340fbb52bca18645a77b7 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:27:18.830264 26072 consensus_queue.cc:1035] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:18.839263 20627 raft_consensus.cc:1261] T ecb8a9b04ba34f779e2227e77e6f764c P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:18.839625 20892 raft_consensus.cc:1261] T ecb8a9b04ba34f779e2227e77e6f764c P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:18.839915 26544 consensus_queue.cc:1035] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:18.840461 26544 consensus_queue.cc:1035] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:18.849025 18699 raft_consensus.cc:1261] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:18.850407 26544 consensus_queue.cc:1035] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:18.855543 26072 raft_consensus.cc:2932] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:18.860539 20760 raft_consensus.cc:2932] T 6acde0e5509340fbb52bca18645a77b7 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:18.867622 18552 catalog_manager.cc:5484] T 6acde0e5509340fbb52bca18645a77b7 P a14c220ad2cc4bf9bd7dad65f4803eb0 reported cstate change: config changed from index 4 to 5, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:18.871632 26844 raft_consensus.cc:2932] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:18.863013 19097 raft_consensus.cc:2932] T 6acde0e5509340fbb52bca18645a77b7 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:18.904464 20627 raft_consensus.cc:2932] T ecb8a9b04ba34f779e2227e77e6f764c P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:18.906699 20894 raft_consensus.cc:2932] T ecb8a9b04ba34f779e2227e77e6f764c P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:18.941887 18699 raft_consensus.cc:2932] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:18.947496 18560 catalog_manager.cc:5484] T ecb8a9b04ba34f779e2227e77e6f764c P a834edffbcb2469aae598b069aab0681 reported cstate change: config changed from index 5 to 6, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:18.947736 18945 tablet_service.cc:1514] Processing DeleteTablet for tablet 6acde0e5509340fbb52bca18645a77b7 with delete_type TABLET_DATA_TOMBSTONED (TS c902805060df4df9b5599d0e94b2dc06 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55752
I20240502 14:27:18.955085 26865 tablet_replica.cc:330] T 6acde0e5509340fbb52bca18645a77b7 P c902805060df4df9b5599d0e94b2dc06: stopping tablet replica
I20240502 14:27:18.955358 26865 raft_consensus.cc:2229] T 6acde0e5509340fbb52bca18645a77b7 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:18.955690 26865 raft_consensus.cc:2258] T 6acde0e5509340fbb52bca18645a77b7 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:18.957083 26865 ts_tablet_manager.cc:1875] T 6acde0e5509340fbb52bca18645a77b7 P c902805060df4df9b5599d0e94b2dc06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:18.962322 18812 tablet_service.cc:1514] Processing DeleteTablet for tablet ecb8a9b04ba34f779e2227e77e6f764c with delete_type TABLET_DATA_TOMBSTONED (TS fec04c50dc5b49878ffa3c3873df32e4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60970
I20240502 14:27:18.966161 26865 ts_tablet_manager.cc:1888] T 6acde0e5509340fbb52bca18645a77b7 P c902805060df4df9b5599d0e94b2dc06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:27:18.966281 26865 log.cc:1196] T 6acde0e5509340fbb52bca18645a77b7 P c902805060df4df9b5599d0e94b2dc06: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal/wals/6acde0e5509340fbb52bca18645a77b7
I20240502 14:27:18.967175 18546 catalog_manager.cc:4830] TS c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409): tablet 6acde0e5509340fbb52bca18645a77b7 (table rebalance_test_table_1 [id=3670a8bef9694c029d2128006cc913d3]) successfully deleted
I20240502 14:27:18.971650 26869 tablet_replica.cc:330] T ecb8a9b04ba34f779e2227e77e6f764c P fec04c50dc5b49878ffa3c3873df32e4: stopping tablet replica
I20240502 14:27:18.977519 26869 raft_consensus.cc:2229] T ecb8a9b04ba34f779e2227e77e6f764c P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:18.978610 26869 raft_consensus.cc:2258] T ecb8a9b04ba34f779e2227e77e6f764c P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:18.980265 26869 ts_tablet_manager.cc:1875] T ecb8a9b04ba34f779e2227e77e6f764c P fec04c50dc5b49878ffa3c3873df32e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:18.993077 26869 ts_tablet_manager.cc:1888] T ecb8a9b04ba34f779e2227e77e6f764c P fec04c50dc5b49878ffa3c3873df32e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:27:18.993256 26869 log.cc:1196] T ecb8a9b04ba34f779e2227e77e6f764c P fec04c50dc5b49878ffa3c3873df32e4: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/wals/ecb8a9b04ba34f779e2227e77e6f764c
I20240502 14:27:18.994007 18543 catalog_manager.cc:4830] TS fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009): tablet ecb8a9b04ba34f779e2227e77e6f764c (table rebalance_test_table_3 [id=210510c054044948932a30db816646aa]) successfully deleted
I20240502 14:27:19.244074 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:19.297358 19076 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:19.325876 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:19.336390 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:19.345368 18965 consensus_queue.cc:237] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:27:19.347479 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:19.352963 20760 raft_consensus.cc:1261] T 246a8425400f4890a5264c4ae8a627c5 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:19.353539 26855 consensus_queue.cc:1035] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:19.391320 20892 raft_consensus.cc:1261] T 246a8425400f4890a5264c4ae8a627c5 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:19.394028 26854 consensus_queue.cc:1035] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:19.394150 19097 raft_consensus.cc:1261] T 246a8425400f4890a5264c4ae8a627c5 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:19.407742 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:19.410581 26854 consensus_queue.cc:1035] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:19.411326 26854 raft_consensus.cc:2932] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:19.420437 20760 raft_consensus.cc:2932] T 246a8425400f4890a5264c4ae8a627c5 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:19.428822 18556 catalog_manager.cc:5484] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 reported cstate change: config changed from index 5 to 6, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:19.433130 18681 tablet_service.cc:1514] Processing DeleteTablet for tablet 246a8425400f4890a5264c4ae8a627c5 with delete_type TABLET_DATA_TOMBSTONED (TS 2db304b538e14857b7b1cc272d0db94b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54454
I20240502 14:27:19.450508 26964 tablet_replica.cc:330] T 246a8425400f4890a5264c4ae8a627c5 P 2db304b538e14857b7b1cc272d0db94b: stopping tablet replica
I20240502 14:27:19.451257 26964 raft_consensus.cc:2229] T 246a8425400f4890a5264c4ae8a627c5 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:19.453347 26964 raft_consensus.cc:2258] T 246a8425400f4890a5264c4ae8a627c5 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:19.453629 20892 raft_consensus.cc:2932] T 246a8425400f4890a5264c4ae8a627c5 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:19.455235 26964 ts_tablet_manager.cc:1875] T 246a8425400f4890a5264c4ae8a627c5 P 2db304b538e14857b7b1cc272d0db94b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:19.462373 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:19.483220 26964 ts_tablet_manager.cc:1888] T 246a8425400f4890a5264c4ae8a627c5 P 2db304b538e14857b7b1cc272d0db94b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:27:19.489706 19097 raft_consensus.cc:2932] T 246a8425400f4890a5264c4ae8a627c5 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:19.499766 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:19.501912 26964 log.cc:1196] T 246a8425400f4890a5264c4ae8a627c5 P 2db304b538e14857b7b1cc272d0db94b: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal/wals/246a8425400f4890a5264c4ae8a627c5
I20240502 14:27:19.503078 18544 catalog_manager.cc:4830] TS 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749): tablet 246a8425400f4890a5264c4ae8a627c5 (table rebalance_test_table_9 [id=a9acd50a500c4f84a41c8d9ba23ae431]) successfully deleted
I20240502 14:27:19.519058 18699 consensus_queue.cc:237] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:27:19.549619 20892 raft_consensus.cc:1261] T b315327141954b53a0f87c2d1dc14b9b P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:27:19.550949 20760 raft_consensus.cc:1261] T b315327141954b53a0f87c2d1dc14b9b P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:27:19.551028 21028 raft_consensus.cc:1261] T b315327141954b53a0f87c2d1dc14b9b P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:27:19.552238 26824 consensus_queue.cc:1035] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:27:19.552511 26980 consensus_queue.cc:1035] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:27:19.552760 26589 consensus_queue.cc:1035] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:27:19.567277 26589 raft_consensus.cc:2932] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:19.577015 20760 raft_consensus.cc:2932] T b315327141954b53a0f87c2d1dc14b9b P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:19.593693 20892 raft_consensus.cc:2932] T b315327141954b53a0f87c2d1dc14b9b P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:19.594647 21028 raft_consensus.cc:2932] T b315327141954b53a0f87c2d1dc14b9b P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:19.593027 18553 catalog_manager.cc:5484] T b315327141954b53a0f87c2d1dc14b9b P a14c220ad2cc4bf9bd7dad65f4803eb0 reported cstate change: config changed from index 7 to 8, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:19.605846 18812 tablet_service.cc:1514] Processing DeleteTablet for tablet b315327141954b53a0f87c2d1dc14b9b with delete_type TABLET_DATA_TOMBSTONED (TS fec04c50dc5b49878ffa3c3873df32e4 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:60970
I20240502 14:27:19.624960 26989 tablet_replica.cc:330] T b315327141954b53a0f87c2d1dc14b9b P fec04c50dc5b49878ffa3c3873df32e4: stopping tablet replica
I20240502 14:27:19.627971 26989 raft_consensus.cc:2229] T b315327141954b53a0f87c2d1dc14b9b P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:19.628269 26989 raft_consensus.cc:2258] T b315327141954b53a0f87c2d1dc14b9b P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:19.629057 26989 ts_tablet_manager.cc:1875] T b315327141954b53a0f87c2d1dc14b9b P fec04c50dc5b49878ffa3c3873df32e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:19.647564 26989 ts_tablet_manager.cc:1888] T b315327141954b53a0f87c2d1dc14b9b P fec04c50dc5b49878ffa3c3873df32e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 14:27:19.651620 26989 log.cc:1196] T b315327141954b53a0f87c2d1dc14b9b P fec04c50dc5b49878ffa3c3873df32e4: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/wals/b315327141954b53a0f87c2d1dc14b9b
I20240502 14:27:19.652673 18543 catalog_manager.cc:4830] TS fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009): tablet b315327141954b53a0f87c2d1dc14b9b (table rebalance_test_table_6 [id=995bc008b9f94332bfa2c42a77e41bb2]) successfully deleted
I20240502 14:27:19.873031 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:19.878130 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:19.896296 19076 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:19.923916 20964 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.68:41463, user_credentials={real_user=slave}} blocked reactor thread for 67993us
I20240502 14:27:19.926839 19076 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:19.929404 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:19.929888 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:19.947124 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:19.970696 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:19.980365 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:20.001832 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:20.007493 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:20.021339 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:20.039577 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:20.078136 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:20.089571 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:20.110023 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:20.330102 19076 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:20.365031 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:20.366405 20606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:27:20.419962 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:20.424657 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:20.429373 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:27:20.439819 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:20.457908 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:20.719139 18832 consensus_queue.cc:237] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:27:20.732156 20760 raft_consensus.cc:1261] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:27:20.732142 19097 raft_consensus.cc:1261] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:27:20.732834 26856 consensus_queue.cc:1035] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:20.733395 27123 consensus_queue.cc:1035] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:20.780596 26856 raft_consensus.cc:2932] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:20.782090 20759 raft_consensus.cc:2932] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:20.796056 18556 catalog_manager.cc:5484] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index 4 to 5, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:20.809803 19097 raft_consensus.cc:2932] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:20.854125 18945 tablet_service.cc:1514] Processing DeleteTablet for tablet 5be0d6f4f1874c79b906c4bdc34b0ce9 with delete_type TABLET_DATA_TOMBSTONED (TS c902805060df4df9b5599d0e94b2dc06 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55752
I20240502 14:27:20.863464 18699 consensus_queue.cc:237] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:27:20.868742 18701 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "522014370f0a4cc3bd31aa00d6a05c5e"
dest_uuid: "2db304b538e14857b7b1cc272d0db94b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35524
I20240502 14:27:20.869473 27185 tablet_replica.cc:330] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06: stopping tablet replica
I20240502 14:27:20.869684 27185 raft_consensus.cc:2229] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:20.869905 27185 raft_consensus.cc:2258] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:20.872052 27185 ts_tablet_manager.cc:1875] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:20.883375 27185 ts_tablet_manager.cc:1888] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:27:20.883525 27185 log.cc:1196] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal/wals/5be0d6f4f1874c79b906c4bdc34b0ce9
I20240502 14:27:20.884212 18546 catalog_manager.cc:4830] TS c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409): tablet 5be0d6f4f1874c79b906c4bdc34b0ce9 (table rebalance_test_table_7 [id=b260896c8b8744c0a1b4f691850aec93]) successfully deleted
I20240502 14:27:20.887112 18701 raft_consensus.cc:595] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 2 LEADER]: Received request to transfer leadership
I20240502 14:27:20.888398 20626 raft_consensus.cc:1261] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [term 2 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:27:20.888680 21028 raft_consensus.cc:1261] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f [term 2 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:27:20.888964 26981 consensus_queue.cc:1035] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:27:20.889340 26824 consensus_queue.cc:1035] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:27:20.908181 26981 raft_consensus.cc:979] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b: : Instructing follower a834edffbcb2469aae598b069aab0681 to start an election
I20240502 14:27:20.908687 26981 raft_consensus.cc:1067] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 2 LEADER]: Signalling peer a834edffbcb2469aae598b069aab0681 to start an election
I20240502 14:27:20.909032 26981 raft_consensus.cc:2932] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:20.909335 21028 raft_consensus.cc:2932] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:20.911667 20627 raft_consensus.cc:2932] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:20.913311 20625 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "522014370f0a4cc3bd31aa00d6a05c5e"
dest_uuid: "a834edffbcb2469aae598b069aab0681"
from {username='slave'} at 127.5.107.65:53335
I20240502 14:27:20.916635 18554 catalog_manager.cc:5484] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f reported cstate change: config changed from index 7 to 8, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New cstate: current_term: 2 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:20.924010 19076 tablet_service.cc:1514] Processing DeleteTablet for tablet 522014370f0a4cc3bd31aa00d6a05c5e with delete_type TABLET_DATA_TOMBSTONED (TS 007b2e79dc424a6f80141180f072f2d8 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:41468
I20240502 14:27:20.924548 20625 raft_consensus.cc:482] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:27:20.924659 20625 raft_consensus.cc:3037] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:27:20.929901 20625 raft_consensus.cc:504] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:27:20.932469 18701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "522014370f0a4cc3bd31aa00d6a05c5e" candidate_uuid: "a834edffbcb2469aae598b069aab0681" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: true dest_uuid: "2db304b538e14857b7b1cc272d0db94b"
I20240502 14:27:20.932749 18701 raft_consensus.cc:3032] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:27:20.932825 18701 raft_consensus.cc:729] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Running, Role: LEADER
I20240502 14:27:20.933126 18701 consensus_queue.cc:260] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:27:20.933457 18701 raft_consensus.cc:3037] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Advancing to term 3
W20240502 14:27:20.936069 26981 consensus_queue.cc:1159] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a834edffbcb2469aae598b069aab0681" responder_term: 2 status { last_received { term: 2 index: 8 } last_committed_idx: 8 last_received_current_leader { term: 2 index: 8 } }
I20240502 14:27:20.937126 18701 raft_consensus.cc:2454] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a834edffbcb2469aae598b069aab0681 in term 3.
I20240502 14:27:20.937827 20560 leader_election.cc:302] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b, a834edffbcb2469aae598b069aab0681; no voters:
I20240502 14:27:20.938140 25149 raft_consensus.cc:2781] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:27:20.951361 21029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "522014370f0a4cc3bd31aa00d6a05c5e" candidate_uuid: "a834edffbcb2469aae598b069aab0681" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: true dest_uuid: "ff6acba046404ef0a6f41ad5e09d852f"
I20240502 14:27:20.951649 21029 raft_consensus.cc:3037] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:27:20.955019 21029 raft_consensus.cc:2454] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a834edffbcb2469aae598b069aab0681 in term 3.
I20240502 14:27:20.958186 20625 leader_election.cc:288] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [CANDIDATE]: Term 3 election: Requested vote from peers 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749), ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871)
I20240502 14:27:20.973527 25149 raft_consensus.cc:686] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [term 3 LEADER]: Becoming Leader. State: Replica: a834edffbcb2469aae598b069aab0681, State: Running, Role: LEADER
I20240502 14:27:20.974067 25149 consensus_queue.cc:237] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:27:20.979811 18556 catalog_manager.cc:5484] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 reported cstate change: term changed from 2 to 3, leader changed from 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) to a834edffbcb2469aae598b069aab0681 (127.5.107.69). New cstate: current_term: 3 leader_uuid: "a834edffbcb2469aae598b069aab0681" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:20.990795 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:20.992764 27201 tablet_replica.cc:330] T 522014370f0a4cc3bd31aa00d6a05c5e P 007b2e79dc424a6f80141180f072f2d8: stopping tablet replica
I20240502 14:27:20.992964 27201 raft_consensus.cc:2229] T 522014370f0a4cc3bd31aa00d6a05c5e P 007b2e79dc424a6f80141180f072f2d8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:20.993155 27201 raft_consensus.cc:2258] T 522014370f0a4cc3bd31aa00d6a05c5e P 007b2e79dc424a6f80141180f072f2d8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:20.993842 27201 ts_tablet_manager.cc:1875] T 522014370f0a4cc3bd31aa00d6a05c5e P 007b2e79dc424a6f80141180f072f2d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:21.045027 19076 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:21.053030 27201 ts_tablet_manager.cc:1888] T 522014370f0a4cc3bd31aa00d6a05c5e P 007b2e79dc424a6f80141180f072f2d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 14:27:21.054853 27201 log.cc:1196] T 522014370f0a4cc3bd31aa00d6a05c5e P 007b2e79dc424a6f80141180f072f2d8: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal/wals/522014370f0a4cc3bd31aa00d6a05c5e
I20240502 14:27:21.055927 18546 catalog_manager.cc:4830] TS 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463): tablet 522014370f0a4cc3bd31aa00d6a05c5e (table rebalance_test_table_2 [id=0e84c82d60e64dbd9149c5f9fb7ff3c8]) successfully deleted
I20240502 14:27:21.065052 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:21.132761 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:21.198287 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:21.225286 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:21.254983 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:21.324035 18897 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.66:33009, user_credentials={real_user=slave}} blocked reactor thread for 105842us
I20240502 14:27:21.326711 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:21.417055 21028 raft_consensus.cc:1261] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f [term 3 FOLLOWER]: Refusing update from remote peer a834edffbcb2469aae598b069aab0681: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240502 14:27:21.417860 25149 consensus_queue.cc:1035] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:27:21.463589 18701 raft_consensus.cc:1261] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 3 FOLLOWER]: Refusing update from remote peer a834edffbcb2469aae598b069aab0681: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240502 14:27:21.464478 27208 consensus_queue.cc:1035] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:27:21.599246 19076 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:21.678686 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:21.688594 19076 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:21.722385 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:21.725822 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:21.757050 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:21.779507 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:21.789566 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:21.810709 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:21.840052 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:21.848361 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:21.859970 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:21.865497 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:21.868459 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:21.869427 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:21.884179 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:22.100450 18701 consensus_queue.cc:237] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:27:22.127347 20626 raft_consensus.cc:1261] T d28db16c9f7e405b8866f0e6c548e246 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:22.128085 27307 consensus_queue.cc:1035] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:22.176010 20626 raft_consensus.cc:2932] T d28db16c9f7e405b8866f0e6c548e246 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:22.185763 27307 raft_consensus.cc:2932] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:22.196696 18556 catalog_manager.cc:5484] T d28db16c9f7e405b8866f0e6c548e246 P a834edffbcb2469aae598b069aab0681 reported cstate change: config changed from index 5 to 6, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
W20240502 14:27:22.223807 18766 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.70:46021, user_credentials={real_user=slave}} blocked reactor thread for 61221.9us
I20240502 14:27:22.275476 18812 tablet_service.cc:1514] Processing DeleteTablet for tablet d28db16c9f7e405b8866f0e6c548e246 with delete_type TABLET_DATA_TOMBSTONED (TS fec04c50dc5b49878ffa3c3873df32e4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60970
I20240502 14:27:22.276031 18701 consensus_queue.cc:237] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
W20240502 14:27:22.279906 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d28db16c9f7e405b8866f0e6c548e246. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:27:22.279925 20626 raft_consensus.cc:1261] T d28db16c9f7e405b8866f0e6c548e246 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:27:22.280589 27392 consensus_queue.cc:1035] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:22.283656 27392 raft_consensus.cc:2932] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:22.292601 20626 raft_consensus.cc:2932] T d28db16c9f7e405b8866f0e6c548e246 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:22.301946 18554 catalog_manager.cc:5484] T d28db16c9f7e405b8866f0e6c548e246 P a834edffbcb2469aae598b069aab0681 reported cstate change: config changed from index 6 to 7, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:22.305241 18544 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet d28db16c9f7e405b8866f0e6c548e246 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:27:22.310448 27421 tablet_replica.cc:330] T d28db16c9f7e405b8866f0e6c548e246 P fec04c50dc5b49878ffa3c3873df32e4: stopping tablet replica
I20240502 14:27:22.311048 27421 raft_consensus.cc:2229] T d28db16c9f7e405b8866f0e6c548e246 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:22.315145 27421 raft_consensus.cc:2258] T d28db16c9f7e405b8866f0e6c548e246 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:22.328047 27421 ts_tablet_manager.cc:1875] T d28db16c9f7e405b8866f0e6c548e246 P fec04c50dc5b49878ffa3c3873df32e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:22.363279 27421 ts_tablet_manager.cc:1888] T d28db16c9f7e405b8866f0e6c548e246 P fec04c50dc5b49878ffa3c3873df32e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:27:22.364204 27421 log.cc:1196] T d28db16c9f7e405b8866f0e6c548e246 P fec04c50dc5b49878ffa3c3873df32e4: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/wals/d28db16c9f7e405b8866f0e6c548e246
I20240502 14:27:22.364995 18543 catalog_manager.cc:4830] TS fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009): tablet d28db16c9f7e405b8866f0e6c548e246 (table rebalance_test_table_0 [id=27066b01c29c4a1380ee4753a1c036d3]) successfully deleted
I20240502 14:27:22.484478 19076 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:22.491120 18965 consensus_queue.cc:237] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:27:22.498853 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:22.518823 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:22.544800 20760 raft_consensus.cc:1261] T 21af917743444f2db932636a78f33512 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:22.545500 27249 consensus_queue.cc:1035] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:22.556629 27249 raft_consensus.cc:2932] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:22.562140 20759 raft_consensus.cc:2932] T 21af917743444f2db932636a78f33512 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:22.568676 18556 catalog_manager.cc:5484] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 reported cstate change: config changed from index 5 to 6, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:22.575237 19076 tablet_service.cc:1514] Processing DeleteTablet for tablet 21af917743444f2db932636a78f33512 with delete_type TABLET_DATA_TOMBSTONED (TS 007b2e79dc424a6f80141180f072f2d8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41468
I20240502 14:27:22.589582 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:22.597996 18965 consensus_queue.cc:237] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:27:22.603866 20760 raft_consensus.cc:1261] T 21af917743444f2db932636a78f33512 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:27:22.604489 27249 consensus_queue.cc:1035] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:22.607059 27489 tablet_replica.cc:330] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8: stopping tablet replica
I20240502 14:27:22.607983 27489 raft_consensus.cc:2229] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:22.608292 27489 raft_consensus.cc:2258] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:22.609206 27489 ts_tablet_manager.cc:1875] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:22.628295 27249 raft_consensus.cc:2932] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:22.631242 20759 raft_consensus.cc:2932] T 21af917743444f2db932636a78f33512 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:22.638615 18554 catalog_manager.cc:5484] T 21af917743444f2db932636a78f33512 P a14c220ad2cc4bf9bd7dad65f4803eb0 reported cstate change: config changed from index 6 to 7, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:22.643234 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:22.646221 18546 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 21af917743444f2db932636a78f33512 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:27:22.660039 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:22.687976 27501 ts_tablet_manager.cc:902] T d28db16c9f7e405b8866f0e6c548e246 P 7ab6d395b30d4afb9821b61c9454430b: Initiating tablet copy from peer 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:27:22.694345 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:27:22.739706 18900 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21af917743444f2db932636a78f33512. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:27:22.740756 20626 consensus_queue.cc:237] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.9, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:27:22.742797 21028 raft_consensus.cc:1261] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f [term 3 FOLLOWER]: Refusing update from remote peer a834edffbcb2469aae598b069aab0681: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 14:27:22.744577 25149 consensus_queue.cc:1035] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:27:22.747912 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:22.769657 27489 ts_tablet_manager.cc:1888] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:27:22.769824 27489 log.cc:1196] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal/wals/21af917743444f2db932636a78f33512
I20240502 14:27:22.773947 18546 catalog_manager.cc:4830] TS 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463): tablet 21af917743444f2db932636a78f33512 (table rebalance_test_table_8 [id=5a8a0451cdf0479c94edf1d94e77fb83]) successfully deleted
I20240502 14:27:22.781363 27501 tablet_copy_client.cc:308] T d28db16c9f7e405b8866f0e6c548e246 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.65:36749
I20240502 14:27:22.782248 18721 tablet_copy_service.cc:140] P 2db304b538e14857b7b1cc272d0db94b: Received BeginTabletCopySession request for tablet d28db16c9f7e405b8866f0e6c548e246 from peer 7ab6d395b30d4afb9821b61c9454430b ({username='slave'} at 127.5.107.71:40777)
I20240502 14:27:22.782403 18721 tablet_copy_service.cc:161] P 2db304b538e14857b7b1cc272d0db94b: Beginning new tablet copy session on tablet d28db16c9f7e405b8866f0e6c548e246 from peer 7ab6d395b30d4afb9821b61c9454430b at {username='slave'} at 127.5.107.71:40777: session id = 7ab6d395b30d4afb9821b61c9454430b-d28db16c9f7e405b8866f0e6c548e246
I20240502 14:27:22.785905 18721 tablet_copy_source_session.cc:205] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:27:22.787487 27501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d28db16c9f7e405b8866f0e6c548e246. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:27:22.828328 27521 raft_consensus.cc:2932] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [term 3 LEADER]: Committing config change with OpId 3.10: config changed from index 8 to 10, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:22.840368 27501 tablet_copy_client.cc:783] T d28db16c9f7e405b8866f0e6c548e246 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Starting download of 0 data blocks...
I20240502 14:27:22.840641 21029 raft_consensus.cc:2932] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f [term 3 FOLLOWER]: Committing config change with OpId 3.10: config changed from index 8 to 10, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:22.841545 27501 tablet_copy_client.cc:647] T d28db16c9f7e405b8866f0e6c548e246 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:27:22.845999 18554 catalog_manager.cc:5484] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f reported cstate change: config changed from index 8 to 10, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New cstate: current_term: 3 leader_uuid: "a834edffbcb2469aae598b069aab0681" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:22.846366 27501 tablet_copy_client.cc:523] T d28db16c9f7e405b8866f0e6c548e246 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:27:22.849953 18681 tablet_service.cc:1514] Processing DeleteTablet for tablet 522014370f0a4cc3bd31aa00d6a05c5e with delete_type TABLET_DATA_TOMBSTONED (TS 2db304b538e14857b7b1cc272d0db94b not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:54454
I20240502 14:27:22.870020 27501 tablet_bootstrap.cc:492] T d28db16c9f7e405b8866f0e6c548e246 P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap starting.
I20240502 14:27:22.882715 27537 tablet_replica.cc:330] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b: stopping tablet replica
I20240502 14:27:22.882990 27537 raft_consensus.cc:2229] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:22.883335 27537 raft_consensus.cc:2258] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:22.884119 27537 ts_tablet_manager.cc:1875] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:22.956270 20626 consensus_queue.cc:237] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:27:22.965482 27501 tablet_bootstrap.cc:492] T d28db16c9f7e405b8866f0e6c548e246 P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:27:22.965562 21029 raft_consensus.cc:1261] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f [term 3 FOLLOWER]: Refusing update from remote peer a834edffbcb2469aae598b069aab0681: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 14:27:22.965938 27501 tablet_bootstrap.cc:492] T d28db16c9f7e405b8866f0e6c548e246 P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap complete.
I20240502 14:27:22.966217 27501 ts_tablet_manager.cc:1367] T d28db16c9f7e405b8866f0e6c548e246 P 7ab6d395b30d4afb9821b61c9454430b: Time spent bootstrapping tablet: real 0.096s user 0.060s sys 0.004s
I20240502 14:27:22.966411 27536 consensus_queue.cc:1035] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:27:22.967222 27501 raft_consensus.cc:348] T d28db16c9f7e405b8866f0e6c548e246 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:27:22.967485 27501 raft_consensus.cc:729] T d28db16c9f7e405b8866f0e6c548e246 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab6d395b30d4afb9821b61c9454430b, State: Initialized, Role: FOLLOWER
I20240502 14:27:22.968374 27501 consensus_queue.cc:260] T d28db16c9f7e405b8866f0e6c548e246 P 7ab6d395b30d4afb9821b61c9454430b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:27:22.969748 27521 raft_consensus.cc:2932] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:22.970337 21029 raft_consensus.cc:2932] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:22.978739 18556 catalog_manager.cc:5484] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f reported cstate change: config changed from index 10 to 11, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New cstate: current_term: 3 leader_uuid: "a834edffbcb2469aae598b069aab0681" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
W20240502 14:27:22.988660 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 522014370f0a4cc3bd31aa00d6a05c5e. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:27:23.009174 18546 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 522014370f0a4cc3bd31aa00d6a05c5e with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:27:23.011687 27537 ts_tablet_manager.cc:1888] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.9
I20240502 14:27:23.011965 27537 log.cc:1196] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal/wals/522014370f0a4cc3bd31aa00d6a05c5e
I20240502 14:27:23.013010 18544 catalog_manager.cc:4830] TS 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749): tablet 522014370f0a4cc3bd31aa00d6a05c5e (table rebalance_test_table_2 [id=0e84c82d60e64dbd9149c5f9fb7ff3c8]) successfully deleted
I20240502 14:27:23.032483 27501 ts_tablet_manager.cc:1398] T d28db16c9f7e405b8866f0e6c548e246 P 7ab6d395b30d4afb9821b61c9454430b: Time spent starting tablet: real 0.066s user 0.027s sys 0.038s
I20240502 14:27:23.033336 18721 tablet_copy_service.cc:342] P 2db304b538e14857b7b1cc272d0db94b: Request end of tablet copy session 7ab6d395b30d4afb9821b61c9454430b-d28db16c9f7e405b8866f0e6c548e246 received from {username='slave'} at 127.5.107.71:40777
I20240502 14:27:23.033450 18721 tablet_copy_service.cc:433] P 2db304b538e14857b7b1cc272d0db94b: ending tablet copy session 7ab6d395b30d4afb9821b61c9454430b-d28db16c9f7e405b8866f0e6c548e246 on tablet d28db16c9f7e405b8866f0e6c548e246 with peer 7ab6d395b30d4afb9821b61c9454430b
I20240502 14:27:23.082933 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:23.146173 19076 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:23.147166 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:23.179633 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:23.180332 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:23.185405 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:23.206508 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:23.209484 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:23.267175 27599 ts_tablet_manager.cc:902] T 21af917743444f2db932636a78f33512 P a834edffbcb2469aae598b069aab0681: Initiating tablet copy from peer c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:27:23.298111 19076 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:23.302516 20894 raft_consensus.cc:1203] T d28db16c9f7e405b8866f0e6c548e246 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:27:23.320497 27599 tablet_copy_client.cc:308] T 21af917743444f2db932636a78f33512 P a834edffbcb2469aae598b069aab0681: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.67:45409
I20240502 14:27:23.321314 18985 tablet_copy_service.cc:140] P c902805060df4df9b5599d0e94b2dc06: Received BeginTabletCopySession request for tablet 21af917743444f2db932636a78f33512 from peer a834edffbcb2469aae598b069aab0681 ({username='slave'} at 127.5.107.69:33739)
I20240502 14:27:23.321444 18985 tablet_copy_service.cc:161] P c902805060df4df9b5599d0e94b2dc06: Beginning new tablet copy session on tablet 21af917743444f2db932636a78f33512 from peer a834edffbcb2469aae598b069aab0681 at {username='slave'} at 127.5.107.69:33739: session id = a834edffbcb2469aae598b069aab0681-21af917743444f2db932636a78f33512
I20240502 14:27:23.323972 18985 tablet_copy_source_session.cc:205] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:27:23.336506 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:23.350137 27599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21af917743444f2db932636a78f33512. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:27:23.368235 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:23.368844 27599 tablet_copy_client.cc:783] T 21af917743444f2db932636a78f33512 P a834edffbcb2469aae598b069aab0681: tablet copy: Starting download of 0 data blocks...
I20240502 14:27:23.369107 27599 tablet_copy_client.cc:647] T 21af917743444f2db932636a78f33512 P a834edffbcb2469aae598b069aab0681: tablet copy: Starting download of 1 WAL segments...
I20240502 14:27:23.386446 27599 tablet_copy_client.cc:523] T 21af917743444f2db932636a78f33512 P a834edffbcb2469aae598b069aab0681: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:27:23.404906 27599 tablet_bootstrap.cc:492] T 21af917743444f2db932636a78f33512 P a834edffbcb2469aae598b069aab0681: Bootstrap starting.
I20240502 14:27:23.470449 27636 ts_tablet_manager.cc:902] T 522014370f0a4cc3bd31aa00d6a05c5e P a14c220ad2cc4bf9bd7dad65f4803eb0: Initiating tablet copy from peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119)
I20240502 14:27:23.475170 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:23.512121 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:23.513092 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:23.514899 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:23.540053 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:23.546263 27636 tablet_copy_client.cc:308] T 522014370f0a4cc3bd31aa00d6a05c5e P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.69:41119
I20240502 14:27:23.617170 18965 consensus_queue.cc:237] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:27:23.634894 20626 raft_consensus.cc:1261] T b13bcd6e693f4c7687a1aabdafbb83c6 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:23.635681 27256 consensus_queue.cc:1035] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:23.636854 20647 tablet_copy_service.cc:140] P a834edffbcb2469aae598b069aab0681: Received BeginTabletCopySession request for tablet 522014370f0a4cc3bd31aa00d6a05c5e from peer a14c220ad2cc4bf9bd7dad65f4803eb0 ({username='slave'} at 127.5.107.70:33125)
I20240502 14:27:23.637023 20647 tablet_copy_service.cc:161] P a834edffbcb2469aae598b069aab0681: Beginning new tablet copy session on tablet 522014370f0a4cc3bd31aa00d6a05c5e from peer a14c220ad2cc4bf9bd7dad65f4803eb0 at {username='slave'} at 127.5.107.70:33125: session id = a14c220ad2cc4bf9bd7dad65f4803eb0-522014370f0a4cc3bd31aa00d6a05c5e
I20240502 14:27:23.641125 20647 tablet_copy_source_session.cc:205] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:27:23.648737 27636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 522014370f0a4cc3bd31aa00d6a05c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:27:23.663897 27599 tablet_bootstrap.cc:492] T 21af917743444f2db932636a78f33512 P a834edffbcb2469aae598b069aab0681: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:27:23.664436 27599 tablet_bootstrap.cc:492] T 21af917743444f2db932636a78f33512 P a834edffbcb2469aae598b069aab0681: Bootstrap complete.
I20240502 14:27:23.664589 27636 tablet_copy_client.cc:783] T 522014370f0a4cc3bd31aa00d6a05c5e P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Starting download of 0 data blocks...
I20240502 14:27:23.664772 27636 tablet_copy_client.cc:647] T 522014370f0a4cc3bd31aa00d6a05c5e P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Starting download of 1 WAL segments...
I20240502 14:27:23.664788 27599 ts_tablet_manager.cc:1367] T 21af917743444f2db932636a78f33512 P a834edffbcb2469aae598b069aab0681: Time spent bootstrapping tablet: real 0.260s user 0.069s sys 0.018s
I20240502 14:27:23.666416 27599 raft_consensus.cc:348] T 21af917743444f2db932636a78f33512 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:27:23.666757 27599 raft_consensus.cc:729] T 21af917743444f2db932636a78f33512 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a834edffbcb2469aae598b069aab0681, State: Initialized, Role: FOLLOWER
I20240502 14:27:23.667093 27599 consensus_queue.cc:260] T 21af917743444f2db932636a78f33512 P a834edffbcb2469aae598b069aab0681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:27:23.668243 27636 tablet_copy_client.cc:523] T 522014370f0a4cc3bd31aa00d6a05c5e P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:27:23.670507 27599 ts_tablet_manager.cc:1398] T 21af917743444f2db932636a78f33512 P a834edffbcb2469aae598b069aab0681: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:27:23.674120 18985 tablet_copy_service.cc:342] P c902805060df4df9b5599d0e94b2dc06: Request end of tablet copy session a834edffbcb2469aae598b069aab0681-21af917743444f2db932636a78f33512 received from {username='slave'} at 127.5.107.69:33739
I20240502 14:27:23.674235 18985 tablet_copy_service.cc:433] P c902805060df4df9b5599d0e94b2dc06: ending tablet copy session a834edffbcb2469aae598b069aab0681-21af917743444f2db932636a78f33512 on tablet 21af917743444f2db932636a78f33512 with peer a834edffbcb2469aae598b069aab0681
I20240502 14:27:23.676108 27636 tablet_bootstrap.cc:492] T 522014370f0a4cc3bd31aa00d6a05c5e P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap starting.
I20240502 14:27:23.736970 27684 raft_consensus.cc:2932] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:23.753131 19097 consensus_queue.cc:237] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:23.755928 20626 raft_consensus.cc:2932] T b13bcd6e693f4c7687a1aabdafbb83c6 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:23.761503 18556 catalog_manager.cc:5484] T b13bcd6e693f4c7687a1aabdafbb83c6 P a834edffbcb2469aae598b069aab0681 reported cstate change: config changed from index 5 to 6, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) evicted. New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:23.782421 21029 raft_consensus.cc:1261] T b15d11209e45473e863766e5cd6e822f P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:27:23.783210 27283 consensus_queue.cc:1035] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:23.802335 27636 tablet_bootstrap.cc:492] T 522014370f0a4cc3bd31aa00d6a05c5e P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:27:23.810448 27636 tablet_bootstrap.cc:492] T 522014370f0a4cc3bd31aa00d6a05c5e P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap complete.
I20240502 14:27:23.811379 18963 raft_consensus.cc:1261] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:27:23.811829 27283 consensus_queue.cc:1035] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:27:23.810911 27636 ts_tablet_manager.cc:1367] T 522014370f0a4cc3bd31aa00d6a05c5e P a14c220ad2cc4bf9bd7dad65f4803eb0: Time spent bootstrapping tablet: real 0.135s user 0.054s sys 0.036s
I20240502 14:27:23.812848 27636 raft_consensus.cc:348] T 522014370f0a4cc3bd31aa00d6a05c5e P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:27:23.813067 27636 raft_consensus.cc:729] T 522014370f0a4cc3bd31aa00d6a05c5e P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: a14c220ad2cc4bf9bd7dad65f4803eb0, State: Initialized, Role: FOLLOWER
I20240502 14:27:23.815105 27636 consensus_queue.cc:260] T 522014370f0a4cc3bd31aa00d6a05c5e P a14c220ad2cc4bf9bd7dad65f4803eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 3.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:27:23.824537 18812 tablet_service.cc:1514] Processing DeleteTablet for tablet b13bcd6e693f4c7687a1aabdafbb83c6 with delete_type TABLET_DATA_TOMBSTONED (TS fec04c50dc5b49878ffa3c3873df32e4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60970
I20240502 14:27:23.825281 27636 ts_tablet_manager.cc:1398] T 522014370f0a4cc3bd31aa00d6a05c5e P a14c220ad2cc4bf9bd7dad65f4803eb0: Time spent starting tablet: real 0.013s user 0.002s sys 0.001s
I20240502 14:27:23.826876 20647 tablet_copy_service.cc:342] P a834edffbcb2469aae598b069aab0681: Request end of tablet copy session a14c220ad2cc4bf9bd7dad65f4803eb0-522014370f0a4cc3bd31aa00d6a05c5e received from {username='slave'} at 127.5.107.70:33125
I20240502 14:27:23.827001 20647 tablet_copy_service.cc:433] P a834edffbcb2469aae598b069aab0681: ending tablet copy session a14c220ad2cc4bf9bd7dad65f4803eb0-522014370f0a4cc3bd31aa00d6a05c5e on tablet 522014370f0a4cc3bd31aa00d6a05c5e with peer a14c220ad2cc4bf9bd7dad65f4803eb0
I20240502 14:27:23.845986 18963 consensus_queue.cc:237] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:27:23.863440 20759 raft_consensus.cc:1203] T 522014370f0a4cc3bd31aa00d6a05c5e P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.10->[3.11-3.11] Dedup: 3.11->[]
I20240502 14:27:23.865803 27721 raft_consensus.cc:2932] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:23.873968 27708 tablet_replica.cc:330] T b13bcd6e693f4c7687a1aabdafbb83c6 P fec04c50dc5b49878ffa3c3873df32e4: stopping tablet replica
I20240502 14:27:23.874177 27708 raft_consensus.cc:2229] T b13bcd6e693f4c7687a1aabdafbb83c6 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:23.874405 27708 raft_consensus.cc:2258] T b13bcd6e693f4c7687a1aabdafbb83c6 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:23.875346 27708 ts_tablet_manager.cc:1875] T b13bcd6e693f4c7687a1aabdafbb83c6 P fec04c50dc5b49878ffa3c3873df32e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:23.875511 20626 raft_consensus.cc:1261] T b13bcd6e693f4c7687a1aabdafbb83c6 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 14:27:23.876188 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b13bcd6e693f4c7687a1aabdafbb83c6. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:27:23.876530 27679 consensus_queue.cc:1035] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:23.885322 21028 raft_consensus.cc:2932] T b15d11209e45473e863766e5cd6e822f P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:23.886438 27679 raft_consensus.cc:2932] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:23.887768 18554 catalog_manager.cc:5484] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: config changed from index 4 to 5, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:23.889640 20626 raft_consensus.cc:2932] T b13bcd6e693f4c7687a1aabdafbb83c6 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:23.895253 18681 tablet_service.cc:1514] Processing DeleteTablet for tablet b15d11209e45473e863766e5cd6e822f with delete_type TABLET_DATA_TOMBSTONED (TS 2db304b538e14857b7b1cc272d0db94b not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54454
I20240502 14:27:23.901336 27708 ts_tablet_manager.cc:1888] T b13bcd6e693f4c7687a1aabdafbb83c6 P fec04c50dc5b49878ffa3c3873df32e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:27:23.902472 27708 log.cc:1196] T b13bcd6e693f4c7687a1aabdafbb83c6 P fec04c50dc5b49878ffa3c3873df32e4: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-1/wal/wals/b13bcd6e693f4c7687a1aabdafbb83c6
I20240502 14:27:23.903407 18543 catalog_manager.cc:4830] TS fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009): tablet b13bcd6e693f4c7687a1aabdafbb83c6 (table rebalance_test_table_4 [id=4dd18a179f134d1babc05959227a25cc]) successfully deleted
I20240502 14:27:23.912077 18554 catalog_manager.cc:5484] T b13bcd6e693f4c7687a1aabdafbb83c6 P a834edffbcb2469aae598b069aab0681 reported cstate change: config changed from index 6 to 7, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:23.912628 18965 raft_consensus.cc:2932] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:23.924523 18546 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet b13bcd6e693f4c7687a1aabdafbb83c6 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:27:23.931562 27731 tablet_replica.cc:330] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b: stopping tablet replica
I20240502 14:27:23.932317 27731 raft_consensus.cc:2229] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:23.935427 27731 raft_consensus.cc:2258] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:23.936933 27731 ts_tablet_manager.cc:1875] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:23.975658 27731 ts_tablet_manager.cc:1888] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:27:23.975781 27731 log.cc:1196] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal/wals/b15d11209e45473e863766e5cd6e822f
I20240502 14:27:23.976656 18544 catalog_manager.cc:4830] TS 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749): tablet b15d11209e45473e863766e5cd6e822f (table rebalance_test_table_5 [id=ff5cff7e35c746a5bc64cc06f18bf7d9]) successfully deleted
I20240502 14:27:24.082640 19076 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:24.092702 20626 raft_consensus.cc:1203] T 21af917743444f2db932636a78f33512 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:27:24.115170 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:24.197911 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:24.201912 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:24.206298 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:24.239295 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:24.294342 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:24.356382 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:24.447280 27850 ts_tablet_manager.cc:902] T b13bcd6e693f4c7687a1aabdafbb83c6 P 7ab6d395b30d4afb9821b61c9454430b: Initiating tablet copy from peer c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:27:24.501118 27850 tablet_copy_client.cc:308] T b13bcd6e693f4c7687a1aabdafbb83c6 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.67:45409
I20240502 14:27:24.501842 18985 tablet_copy_service.cc:140] P c902805060df4df9b5599d0e94b2dc06: Received BeginTabletCopySession request for tablet b13bcd6e693f4c7687a1aabdafbb83c6 from peer 7ab6d395b30d4afb9821b61c9454430b ({username='slave'} at 127.5.107.71:55147)
I20240502 14:27:24.501984 18985 tablet_copy_service.cc:161] P c902805060df4df9b5599d0e94b2dc06: Beginning new tablet copy session on tablet b13bcd6e693f4c7687a1aabdafbb83c6 from peer 7ab6d395b30d4afb9821b61c9454430b at {username='slave'} at 127.5.107.71:55147: session id = 7ab6d395b30d4afb9821b61c9454430b-b13bcd6e693f4c7687a1aabdafbb83c6
I20240502 14:27:24.504599 18985 tablet_copy_source_session.cc:205] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:27:24.506577 27850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b13bcd6e693f4c7687a1aabdafbb83c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:27:24.514425 27850 tablet_copy_client.cc:783] T b13bcd6e693f4c7687a1aabdafbb83c6 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Starting download of 0 data blocks...
I20240502 14:27:24.515293 27850 tablet_copy_client.cc:647] T b13bcd6e693f4c7687a1aabdafbb83c6 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:27:24.517068 27850 tablet_copy_client.cc:523] T b13bcd6e693f4c7687a1aabdafbb83c6 P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:27:24.540618 27850 tablet_bootstrap.cc:492] T b13bcd6e693f4c7687a1aabdafbb83c6 P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap starting.
I20240502 14:27:24.636915 27850 tablet_bootstrap.cc:492] T b13bcd6e693f4c7687a1aabdafbb83c6 P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:27:24.637539 27850 tablet_bootstrap.cc:492] T b13bcd6e693f4c7687a1aabdafbb83c6 P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap complete.
I20240502 14:27:24.637910 27850 ts_tablet_manager.cc:1367] T b13bcd6e693f4c7687a1aabdafbb83c6 P 7ab6d395b30d4afb9821b61c9454430b: Time spent bootstrapping tablet: real 0.097s user 0.048s sys 0.005s
I20240502 14:27:24.659355 27850 raft_consensus.cc:348] T b13bcd6e693f4c7687a1aabdafbb83c6 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:27:24.659723 27850 raft_consensus.cc:729] T b13bcd6e693f4c7687a1aabdafbb83c6 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab6d395b30d4afb9821b61c9454430b, State: Initialized, Role: FOLLOWER
I20240502 14:27:24.659992 27850 consensus_queue.cc:260] T b13bcd6e693f4c7687a1aabdafbb83c6 P 7ab6d395b30d4afb9821b61c9454430b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:27:24.699589 27850 ts_tablet_manager.cc:1398] T b13bcd6e693f4c7687a1aabdafbb83c6 P 7ab6d395b30d4afb9821b61c9454430b: Time spent starting tablet: real 0.041s user 0.004s sys 0.000s
I20240502 14:27:24.705718 18985 tablet_copy_service.cc:342] P c902805060df4df9b5599d0e94b2dc06: Request end of tablet copy session 7ab6d395b30d4afb9821b61c9454430b-b13bcd6e693f4c7687a1aabdafbb83c6 received from {username='slave'} at 127.5.107.71:55147
I20240502 14:27:24.705854 18985 tablet_copy_service.cc:433] P c902805060df4df9b5599d0e94b2dc06: ending tablet copy session 7ab6d395b30d4afb9821b61c9454430b-b13bcd6e693f4c7687a1aabdafbb83c6 on tablet b13bcd6e693f4c7687a1aabdafbb83c6 with peer 7ab6d395b30d4afb9821b61c9454430b
I20240502 14:27:24.712983 19076 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:24.724625 19076 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:24.749368 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:24.762961 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:24.774474 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:24.788765 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:24.805846 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:24.840885 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:24.892495 18832 consensus_queue.cc:237] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:24.926731 21028 raft_consensus.cc:1261] T 8b8d81c275144f898dcc69a6d98d11e0 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:27:24.927287 27390 consensus_queue.cc:1035] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:24.929435 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:24.930554 27972 raft_consensus.cc:2932] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:24.931986 21029 raft_consensus.cc:2932] T 8b8d81c275144f898dcc69a6d98d11e0 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:24.935247 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:24.938895 18554 catalog_manager.cc:5484] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index 6 to 7, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:24.962306 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:24.963716 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:24.975590 20894 raft_consensus.cc:1261] T 8b8d81c275144f898dcc69a6d98d11e0 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:27:24.976614 27390 consensus_queue.cc:1035] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:24.979130 20894 raft_consensus.cc:2932] T 8b8d81c275144f898dcc69a6d98d11e0 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:24.992815 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:25.009770 18681 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b8d81c275144f898dcc69a6d98d11e0 with delete_type TABLET_DATA_TOMBSTONED (TS 2db304b538e14857b7b1cc272d0db94b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54454
I20240502 14:27:25.016041 20894 raft_consensus.cc:1203] T b13bcd6e693f4c7687a1aabdafbb83c6 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:27:25.023934 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:25.028081 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:25.049518 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:25.059144 27990 tablet_replica.cc:330] T 8b8d81c275144f898dcc69a6d98d11e0 P 2db304b538e14857b7b1cc272d0db94b: stopping tablet replica
I20240502 14:27:25.059365 27990 raft_consensus.cc:2229] T 8b8d81c275144f898dcc69a6d98d11e0 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:25.059634 27990 raft_consensus.cc:2258] T 8b8d81c275144f898dcc69a6d98d11e0 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:25.062299 27990 ts_tablet_manager.cc:1875] T 8b8d81c275144f898dcc69a6d98d11e0 P 2db304b538e14857b7b1cc272d0db94b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:25.074208 27990 ts_tablet_manager.cc:1888] T 8b8d81c275144f898dcc69a6d98d11e0 P 2db304b538e14857b7b1cc272d0db94b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:27:25.074322 27990 log.cc:1196] T 8b8d81c275144f898dcc69a6d98d11e0 P 2db304b538e14857b7b1cc272d0db94b: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal/wals/8b8d81c275144f898dcc69a6d98d11e0
I20240502 14:27:25.075146 18544 catalog_manager.cc:4830] TS 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749): tablet 8b8d81c275144f898dcc69a6d98d11e0 (table rebalance_test_table_7 [id=b260896c8b8744c0a1b4f691850aec93]) successfully deleted
I20240502 14:27:25.342311 19093 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b15d11209e45473e863766e5cd6e822f"
dest_uuid: "007b2e79dc424a6f80141180f072f2d8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51746
I20240502 14:27:25.342510 19093 raft_consensus.cc:595] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:27:25.359647 27283 raft_consensus.cc:979] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8: : Instructing follower c902805060df4df9b5599d0e94b2dc06 to start an election
I20240502 14:27:25.359783 27283 raft_consensus.cc:1067] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Signalling peer c902805060df4df9b5599d0e94b2dc06 to start an election
I20240502 14:27:25.361332 18965 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b15d11209e45473e863766e5cd6e822f"
dest_uuid: "c902805060df4df9b5599d0e94b2dc06"
from {username='slave'} at 127.5.107.68:39711
I20240502 14:27:25.361574 18965 raft_consensus.cc:482] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:27:25.361654 18965 raft_consensus.cc:3037] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:27:25.364401 18965 raft_consensus.cc:504] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:25.366686 19093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15d11209e45473e863766e5cd6e822f" candidate_uuid: "c902805060df4df9b5599d0e94b2dc06" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "007b2e79dc424a6f80141180f072f2d8"
I20240502 14:27:25.366886 19093 raft_consensus.cc:3032] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:27:25.366954 19093 raft_consensus.cc:729] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Running, Role: LEADER
I20240502 14:27:25.367228 19093 consensus_queue.cc:260] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:25.367597 19093 raft_consensus.cc:3037] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:27:25.370666 19093 raft_consensus.cc:2454] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c902805060df4df9b5599d0e94b2dc06 in term 2.
I20240502 14:27:25.371078 18900 leader_election.cc:302] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 007b2e79dc424a6f80141180f072f2d8, c902805060df4df9b5599d0e94b2dc06; no voters:
I20240502 14:27:25.413800 18965 leader_election.cc:288] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 2 election: Requested vote from peers 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463), ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871)
I20240502 14:27:25.420526 27679 raft_consensus.cc:2781] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [term 2 FOLLOWER]: Leader election won for term 2
W20240502 14:27:25.429690 18900 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.5.107.68:41463, user_credentials={real_user=slave}} blocked reactor thread for 58634.5us
I20240502 14:27:25.456954 27679 raft_consensus.cc:686] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [term 2 LEADER]: Becoming Leader. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Running, Role: LEADER
I20240502 14:27:25.457269 27679 consensus_queue.cc:237] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:25.460832 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15d11209e45473e863766e5cd6e822f" candidate_uuid: "c902805060df4df9b5599d0e94b2dc06" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ff6acba046404ef0a6f41ad5e09d852f"
I20240502 14:27:25.461015 21028 raft_consensus.cc:3037] T b15d11209e45473e863766e5cd6e822f P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:27:25.464346 21028 raft_consensus.cc:2454] T b15d11209e45473e863766e5cd6e822f P ff6acba046404ef0a6f41ad5e09d852f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c902805060df4df9b5599d0e94b2dc06 in term 2.
I20240502 14:27:25.466046 18699 consensus_queue.cc:237] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:27:25.469087 20759 raft_consensus.cc:1261] T 96662a3d9e1b4f03b05464ddd06b93d6 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:25.469192 20894 raft_consensus.cc:1261] T 96662a3d9e1b4f03b05464ddd06b93d6 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:25.469735 27884 consensus_queue.cc:1035] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:25.470348 18965 raft_consensus.cc:1261] T 96662a3d9e1b4f03b05464ddd06b93d6 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:25.470839 27884 consensus_queue.cc:1035] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:25.471612 28069 consensus_queue.cc:1035] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:25.495659 28069 raft_consensus.cc:2932] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:25.496070 18965 raft_consensus.cc:2932] T 96662a3d9e1b4f03b05464ddd06b93d6 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:25.529886 18560 catalog_manager.cc:5484] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b reported cstate change: config changed from index 5 to 6, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:25.534159 18553 catalog_manager.cc:5484] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 reported cstate change: term changed from 1 to 2, leader changed from 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) to c902805060df4df9b5599d0e94b2dc06 (127.5.107.67). New cstate: current_term: 2 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:25.538136 20759 raft_consensus.cc:2932] T 96662a3d9e1b4f03b05464ddd06b93d6 P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:25.575759 20894 raft_consensus.cc:2932] T 96662a3d9e1b4f03b05464ddd06b93d6 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:25.584112 19076 tablet_service.cc:1514] Processing DeleteTablet for tablet 96662a3d9e1b4f03b05464ddd06b93d6 with delete_type TABLET_DATA_TOMBSTONED (TS 007b2e79dc424a6f80141180f072f2d8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41468
I20240502 14:27:25.584388 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:25.611555 28110 tablet_replica.cc:330] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8: stopping tablet replica
I20240502 14:27:25.616806 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:25.621439 28110 raft_consensus.cc:2229] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:25.621738 28110 raft_consensus.cc:2258] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:25.622611 28110 ts_tablet_manager.cc:1875] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:25.639927 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:25.663955 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:25.667459 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:25.674904 28110 ts_tablet_manager.cc:1888] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:27:25.675900 28110 log.cc:1196] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal/wals/96662a3d9e1b4f03b05464ddd06b93d6
I20240502 14:27:25.682057 18546 catalog_manager.cc:4830] TS 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463): tablet 96662a3d9e1b4f03b05464ddd06b93d6 (table rebalance_test_table_2 [id=0e84c82d60e64dbd9149c5f9fb7ff3c8]) successfully deleted
I20240502 14:27:25.740541 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:25.778888 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:25.792261 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:25.857349 19093 raft_consensus.cc:1261] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [term 2 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:27:25.858194 28082 consensus_queue.cc:1035] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:25.921416 21028 raft_consensus.cc:1261] T b15d11209e45473e863766e5cd6e822f P ff6acba046404ef0a6f41ad5e09d852f [term 2 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:27:25.922283 27679 consensus_queue.cc:1035] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:25.986316 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:25.998625 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:26.015995 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:26.065865 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:26.110816 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:26.133793 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:26.202282 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:26.264873 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:26.284139 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:26.297205 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:26.298419 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:26.342828 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:26.347836 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:26.379009 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:26.401640 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:26.406248 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:26.524048 18832 consensus_queue.cc:237] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:26.535739 21029 raft_consensus.cc:1261] T 2c2afaffde96462eacd0d32bbdec23b1 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:27:26.536180 27972 consensus_queue.cc:1035] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:26.543979 20626 raft_consensus.cc:1261] T 2c2afaffde96462eacd0d32bbdec23b1 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:27:26.544615 27972 consensus_queue.cc:1035] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:26.559994 27972 raft_consensus.cc:2932] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:26.561344 21028 raft_consensus.cc:2932] T 2c2afaffde96462eacd0d32bbdec23b1 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:26.599738 20626 raft_consensus.cc:2932] T 2c2afaffde96462eacd0d32bbdec23b1 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:26.602313 18553 catalog_manager.cc:5484] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index 6 to 7, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:26.647084 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:26.652837 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:26.686749 18945 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c2afaffde96462eacd0d32bbdec23b1 with delete_type TABLET_DATA_TOMBSTONED (TS c902805060df4df9b5599d0e94b2dc06 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55752
I20240502 14:27:26.689108 28310 tablet_replica.cc:330] T 2c2afaffde96462eacd0d32bbdec23b1 P c902805060df4df9b5599d0e94b2dc06: stopping tablet replica
I20240502 14:27:26.691159 28310 raft_consensus.cc:2229] T 2c2afaffde96462eacd0d32bbdec23b1 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:26.691588 28310 raft_consensus.cc:2258] T 2c2afaffde96462eacd0d32bbdec23b1 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:26.692664 28310 ts_tablet_manager.cc:1875] T 2c2afaffde96462eacd0d32bbdec23b1 P c902805060df4df9b5599d0e94b2dc06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:26.715541 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:26.732026 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:26.758572 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:26.769026 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:26.769846 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:26.804788 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:26.804896 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:26.811215 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:26.819473 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:26.840562 28310 ts_tablet_manager.cc:1888] T 2c2afaffde96462eacd0d32bbdec23b1 P c902805060df4df9b5599d0e94b2dc06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:27:26.840757 28310 log.cc:1196] T 2c2afaffde96462eacd0d32bbdec23b1 P c902805060df4df9b5599d0e94b2dc06: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal/wals/2c2afaffde96462eacd0d32bbdec23b1
I20240502 14:27:26.851156 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:26.863386 18546 catalog_manager.cc:4830] TS c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409): tablet 2c2afaffde96462eacd0d32bbdec23b1 (table rebalance_test_table_9 [id=a9acd50a500c4f84a41c8d9ba23ae431]) successfully deleted
I20240502 14:27:26.912276 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:26.930469 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:27.001389 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:27.024605 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:27.227380 20626 consensus_queue.cc:237] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:27:27.239639 21028 raft_consensus.cc:1261] T 51ef0be93c914e258635e13f3cca79aa P ff6acba046404ef0a6f41ad5e09d852f [term 2 FOLLOWER]: Refusing update from remote peer a834edffbcb2469aae598b069aab0681: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:27:27.240332 27798 consensus_queue.cc:1035] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:27:27.266960 18965 consensus_queue.cc:237] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:27.270193 21028 raft_consensus.cc:1261] T b15d11209e45473e863766e5cd6e822f P ff6acba046404ef0a6f41ad5e09d852f [term 2 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:27:27.270752 27679 consensus_queue.cc:1035] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:27.276832 18965 raft_consensus.cc:1261] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06 [term 2 FOLLOWER]: Refusing update from remote peer a834edffbcb2469aae598b069aab0681: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:27:27.277608 27798 consensus_queue.cc:1035] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:27:27.294044 28417 raft_consensus.cc:2932] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:27.295246 21029 raft_consensus.cc:2932] T b15d11209e45473e863766e5cd6e822f P ff6acba046404ef0a6f41ad5e09d852f [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
W20240502 14:27:27.300179 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 51ef0be93c914e258635e13f3cca79aa. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:27:27.300474 18554 catalog_manager.cc:5484] T b15d11209e45473e863766e5cd6e822f P ff6acba046404ef0a6f41ad5e09d852f reported cstate change: config changed from index 5 to 7, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New cstate: current_term: 2 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:27.304042 27798 raft_consensus.cc:2932] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:27.310544 18699 consensus_queue.cc:237] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:27.318372 18965 raft_consensus.cc:2932] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:27.361428 21029 raft_consensus.cc:1261] T db56136f2dbb429098d4d78b15a3a439 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:27:27.362673 20626 raft_consensus.cc:1261] T db56136f2dbb429098d4d78b15a3a439 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:27:27.363050 28329 consensus_queue.cc:1035] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:27.365027 28329 consensus_queue.cc:1035] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:27.375366 21028 raft_consensus.cc:2932] T 51ef0be93c914e258635e13f3cca79aa P ff6acba046404ef0a6f41ad5e09d852f [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:27.382539 18560 catalog_manager.cc:5484] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 reported cstate change: config changed from index 8 to 9, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New cstate: current_term: 2 leader_uuid: "a834edffbcb2469aae598b069aab0681" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:27.386551 19076 tablet_service.cc:1514] Processing DeleteTablet for tablet b15d11209e45473e863766e5cd6e822f with delete_type TABLET_DATA_TOMBSTONED (TS 007b2e79dc424a6f80141180f072f2d8 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:41468
I20240502 14:27:27.392463 18965 consensus_queue.cc:237] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:27:27.393733 28434 tablet_replica.cc:330] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8: stopping tablet replica
W20240502 14:27:27.394639 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b15d11209e45473e863766e5cd6e822f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:27:27.394649 21028 raft_consensus.cc:1261] T b15d11209e45473e863766e5cd6e822f P ff6acba046404ef0a6f41ad5e09d852f [term 2 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:27:27.395316 28434 raft_consensus.cc:2229] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:27.395247 28417 consensus_queue.cc:1035] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:27:27.396253 28434 raft_consensus.cc:2258] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:27.397624 28434 ts_tablet_manager.cc:1875] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:27.400701 28417 raft_consensus.cc:2932] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:27.405769 21028 raft_consensus.cc:2932] T b15d11209e45473e863766e5cd6e822f P ff6acba046404ef0a6f41ad5e09d852f [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:27.408964 18546 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet b15d11209e45473e863766e5cd6e822f with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:27:27.433899 19093 consensus_queue.cc:237] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:27:27.434347 18553 catalog_manager.cc:5484] T b15d11209e45473e863766e5cd6e822f P ff6acba046404ef0a6f41ad5e09d852f reported cstate change: config changed from index 7 to 8, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New cstate: current_term: 2 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:27.442188 28443 raft_consensus.cc:2932] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:27.447436 20625 raft_consensus.cc:2932] T db56136f2dbb429098d4d78b15a3a439 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:27.453402 18553 catalog_manager.cc:5484] T db56136f2dbb429098d4d78b15a3a439 P a834edffbcb2469aae598b069aab0681 reported cstate change: config changed from index 6 to 7, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:27.456190 19076 tablet_service.cc:1514] Processing DeleteTablet for tablet db56136f2dbb429098d4d78b15a3a439 with delete_type TABLET_DATA_TOMBSTONED (TS 007b2e79dc424a6f80141180f072f2d8 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:41468
I20240502 14:27:27.460336 28434 ts_tablet_manager.cc:1888] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 14:27:27.460444 28434 log.cc:1196] T b15d11209e45473e863766e5cd6e822f P 007b2e79dc424a6f80141180f072f2d8: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal/wals/b15d11209e45473e863766e5cd6e822f
I20240502 14:27:27.461167 18546 catalog_manager.cc:4830] TS 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463): tablet b15d11209e45473e863766e5cd6e822f (table rebalance_test_table_5 [id=ff5cff7e35c746a5bc64cc06f18bf7d9]) successfully deleted
I20240502 14:27:27.462232 21029 raft_consensus.cc:2932] T db56136f2dbb429098d4d78b15a3a439 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:27.467727 20626 raft_consensus.cc:1261] T 097eb705c9e24740a5be61bf3354f3d8 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:27.468319 27849 consensus_queue.cc:1035] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:27.478351 28434 tablet_replica.cc:330] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8: stopping tablet replica
I20240502 14:27:27.478617 28434 raft_consensus.cc:2229] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:27.478865 28434 raft_consensus.cc:2258] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:27.479703 28434 ts_tablet_manager.cc:1875] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:27.521584 28469 raft_consensus.cc:2932] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:27.525578 20626 raft_consensus.cc:2932] T 097eb705c9e24740a5be61bf3354f3d8 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:27.530388 18553 catalog_manager.cc:5484] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: config changed from index 5 to 6, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:27.542014 18681 tablet_service.cc:1514] Processing DeleteTablet for tablet 097eb705c9e24740a5be61bf3354f3d8 with delete_type TABLET_DATA_TOMBSTONED (TS 2db304b538e14857b7b1cc272d0db94b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54454
I20240502 14:27:27.542685 28434 ts_tablet_manager.cc:1888] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:27:27.542806 28434 log.cc:1196] T db56136f2dbb429098d4d78b15a3a439 P 007b2e79dc424a6f80141180f072f2d8: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal/wals/db56136f2dbb429098d4d78b15a3a439
I20240502 14:27:27.543656 18546 catalog_manager.cc:4830] TS 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463): tablet db56136f2dbb429098d4d78b15a3a439 (table rebalance_test_table_8 [id=5a8a0451cdf0479c94edf1d94e77fb83]) successfully deleted
I20240502 14:27:27.548067 19093 consensus_queue.cc:237] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:27:27.554283 18699 raft_consensus.cc:1261] T 097eb705c9e24740a5be61bf3354f3d8 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:27:27.554849 27849 consensus_queue.cc:1035] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:27.555063 20626 raft_consensus.cc:1261] T 097eb705c9e24740a5be61bf3354f3d8 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:27:27.555601 28469 consensus_queue.cc:1035] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:27.557415 18699 raft_consensus.cc:858] T 097eb705c9e24740a5be61bf3354f3d8 P 2db304b538e14857b7b1cc272d0db94b [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
W20240502 14:27:27.557498 18699 raft_consensus.cc:1565] T 097eb705c9e24740a5be61bf3354f3d8 P 2db304b538e14857b7b1cc272d0db94b [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:27:27.557933 27849 raft_consensus.cc:2932] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:27:27.561770 20625 raft_consensus.cc:2932] T 097eb705c9e24740a5be61bf3354f3d8 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:27:27.562655 18546 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 097eb705c9e24740a5be61bf3354f3d8 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:27:27.564749 18556 catalog_manager.cc:5484] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: config changed from index 6 to 7, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) added. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:27:27.574326 18699 raft_consensus.cc:2932] T 097eb705c9e24740a5be61bf3354f3d8 P 2db304b538e14857b7b1cc272d0db94b [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:27.579711 18699 raft_consensus.cc:1467] T 097eb705c9e24740a5be61bf3354f3d8 P 2db304b538e14857b7b1cc272d0db94b [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20240502 14:27:27.580245 18699 raft_consensus.cc:2932] T 097eb705c9e24740a5be61bf3354f3d8 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
W20240502 14:27:27.581053 18544 catalog_manager.cc:4804] TS 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749): delete failed for tablet 097eb705c9e24740a5be61bf3354f3d8 because of a CAS failure. No further retry: Illegal state: Request specified cas_config_opid_index_less_or_equal of 5 but the committed config has opid_index of 6
I20240502 14:27:27.669566 28502 ts_tablet_manager.cc:902] T 51ef0be93c914e258635e13f3cca79aa P 7ab6d395b30d4afb9821b61c9454430b: Initiating tablet copy from peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119)
I20240502 14:27:27.732434 28502 tablet_copy_client.cc:308] T 51ef0be93c914e258635e13f3cca79aa P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.69:41119
I20240502 14:27:27.816337 20647 tablet_copy_service.cc:140] P a834edffbcb2469aae598b069aab0681: Received BeginTabletCopySession request for tablet 51ef0be93c914e258635e13f3cca79aa from peer 7ab6d395b30d4afb9821b61c9454430b ({username='slave'} at 127.5.107.71:38265)
I20240502 14:27:27.816534 20647 tablet_copy_service.cc:161] P a834edffbcb2469aae598b069aab0681: Beginning new tablet copy session on tablet 51ef0be93c914e258635e13f3cca79aa from peer 7ab6d395b30d4afb9821b61c9454430b at {username='slave'} at 127.5.107.71:38265: session id = 7ab6d395b30d4afb9821b61c9454430b-51ef0be93c914e258635e13f3cca79aa
I20240502 14:27:27.821746 20647 tablet_copy_source_session.cc:205] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:27:27.823109 28502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51ef0be93c914e258635e13f3cca79aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:27:27.831569 28502 tablet_copy_client.cc:783] T 51ef0be93c914e258635e13f3cca79aa P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Starting download of 0 data blocks...
I20240502 14:27:27.831943 28502 tablet_copy_client.cc:647] T 51ef0be93c914e258635e13f3cca79aa P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:27:27.833225 28502 tablet_copy_client.cc:523] T 51ef0be93c914e258635e13f3cca79aa P 7ab6d395b30d4afb9821b61c9454430b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:27:27.837157 28502 tablet_bootstrap.cc:492] T 51ef0be93c914e258635e13f3cca79aa P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap starting.
I20240502 14:27:27.889933 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:27.901113 28558 ts_tablet_manager.cc:902] T b15d11209e45473e863766e5cd6e822f P a14c220ad2cc4bf9bd7dad65f4803eb0: Initiating tablet copy from peer c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:27:27.954509 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:27.954109 28558 tablet_copy_client.cc:308] T b15d11209e45473e863766e5cd6e822f P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.67:45409
I20240502 14:27:27.956579 18985 tablet_copy_service.cc:140] P c902805060df4df9b5599d0e94b2dc06: Received BeginTabletCopySession request for tablet b15d11209e45473e863766e5cd6e822f from peer a14c220ad2cc4bf9bd7dad65f4803eb0 ({username='slave'} at 127.5.107.70:56629)
I20240502 14:27:27.956735 18985 tablet_copy_service.cc:161] P c902805060df4df9b5599d0e94b2dc06: Beginning new tablet copy session on tablet b15d11209e45473e863766e5cd6e822f from peer a14c220ad2cc4bf9bd7dad65f4803eb0 at {username='slave'} at 127.5.107.70:56629: session id = a14c220ad2cc4bf9bd7dad65f4803eb0-b15d11209e45473e863766e5cd6e822f
I20240502 14:27:27.963222 18985 tablet_copy_source_session.cc:205] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:27:27.965988 28502 tablet_bootstrap.cc:492] T 51ef0be93c914e258635e13f3cca79aa P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:27:27.966346 28502 tablet_bootstrap.cc:492] T 51ef0be93c914e258635e13f3cca79aa P 7ab6d395b30d4afb9821b61c9454430b: Bootstrap complete.
I20240502 14:27:27.966632 28502 ts_tablet_manager.cc:1367] T 51ef0be93c914e258635e13f3cca79aa P 7ab6d395b30d4afb9821b61c9454430b: Time spent bootstrapping tablet: real 0.130s user 0.049s sys 0.004s
I20240502 14:27:27.967281 28502 raft_consensus.cc:348] T 51ef0be93c914e258635e13f3cca79aa P 7ab6d395b30d4afb9821b61c9454430b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:27:27.967521 28502 raft_consensus.cc:729] T 51ef0be93c914e258635e13f3cca79aa P 7ab6d395b30d4afb9821b61c9454430b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab6d395b30d4afb9821b61c9454430b, State: Initialized, Role: FOLLOWER
I20240502 14:27:27.967862 28502 consensus_queue.cc:260] T 51ef0be93c914e258635e13f3cca79aa P 7ab6d395b30d4afb9821b61c9454430b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:27:27.979388 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:27.981608 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:27.996234 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:28.002151 28558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b15d11209e45473e863766e5cd6e822f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:27:28.030239 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:28.055227 28558 tablet_copy_client.cc:783] T b15d11209e45473e863766e5cd6e822f P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Starting download of 0 data blocks...
I20240502 14:27:28.055552 28558 tablet_copy_client.cc:647] T b15d11209e45473e863766e5cd6e822f P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Starting download of 1 WAL segments...
I20240502 14:27:28.057425 28558 tablet_copy_client.cc:523] T b15d11209e45473e863766e5cd6e822f P a14c220ad2cc4bf9bd7dad65f4803eb0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:27:28.060719 28558 tablet_bootstrap.cc:492] T b15d11209e45473e863766e5cd6e822f P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap starting.
I20240502 14:27:28.065300 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:28.100939 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:28.116974 28502 ts_tablet_manager.cc:1398] T 51ef0be93c914e258635e13f3cca79aa P 7ab6d395b30d4afb9821b61c9454430b: Time spent starting tablet: real 0.150s user 0.049s sys 0.043s
I20240502 14:27:28.117883 20647 tablet_copy_service.cc:342] P a834edffbcb2469aae598b069aab0681: Request end of tablet copy session 7ab6d395b30d4afb9821b61c9454430b-51ef0be93c914e258635e13f3cca79aa received from {username='slave'} at 127.5.107.71:38265
I20240502 14:27:28.118016 20647 tablet_copy_service.cc:433] P a834edffbcb2469aae598b069aab0681: ending tablet copy session 7ab6d395b30d4afb9821b61c9454430b-51ef0be93c914e258635e13f3cca79aa on tablet 51ef0be93c914e258635e13f3cca79aa with peer 7ab6d395b30d4afb9821b61c9454430b
I20240502 14:27:28.212658 20892 raft_consensus.cc:1203] T 51ef0be93c914e258635e13f3cca79aa P 7ab6d395b30d4afb9821b61c9454430b [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20240502 14:27:28.386682 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:28.388885 28558 tablet_bootstrap.cc:492] T b15d11209e45473e863766e5cd6e822f P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:27:28.411417 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:28.436748 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:28.451570 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:28.475401 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:28.483942 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:28.492225 28558 tablet_bootstrap.cc:492] T b15d11209e45473e863766e5cd6e822f P a14c220ad2cc4bf9bd7dad65f4803eb0: Bootstrap complete.
I20240502 14:27:28.511858 28558 ts_tablet_manager.cc:1367] T b15d11209e45473e863766e5cd6e822f P a14c220ad2cc4bf9bd7dad65f4803eb0: Time spent bootstrapping tablet: real 0.451s user 0.076s sys 0.002s
I20240502 14:27:28.517884 28558 raft_consensus.cc:348] T b15d11209e45473e863766e5cd6e822f P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:27:28.519109 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:28.519412 28558 raft_consensus.cc:729] T b15d11209e45473e863766e5cd6e822f P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a14c220ad2cc4bf9bd7dad65f4803eb0, State: Initialized, Role: FOLLOWER
I20240502 14:27:28.519889 28558 consensus_queue.cc:260] T b15d11209e45473e863766e5cd6e822f P a14c220ad2cc4bf9bd7dad65f4803eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:27:28.557555 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:28.587630 28558 ts_tablet_manager.cc:1398] T b15d11209e45473e863766e5cd6e822f P a14c220ad2cc4bf9bd7dad65f4803eb0: Time spent starting tablet: real 0.070s user 0.005s sys 0.004s
I20240502 14:27:28.588344 18985 tablet_copy_service.cc:342] P c902805060df4df9b5599d0e94b2dc06: Request end of tablet copy session a14c220ad2cc4bf9bd7dad65f4803eb0-b15d11209e45473e863766e5cd6e822f received from {username='slave'} at 127.5.107.70:56629
I20240502 14:27:28.588511 18985 tablet_copy_service.cc:433] P c902805060df4df9b5599d0e94b2dc06: ending tablet copy session a14c220ad2cc4bf9bd7dad65f4803eb0-b15d11209e45473e863766e5cd6e822f on tablet b15d11209e45473e863766e5cd6e822f with peer a14c220ad2cc4bf9bd7dad65f4803eb0
I20240502 14:27:28.726615 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:28.812178 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:28.819989 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:28.823733 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:28.848968 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:28.852191 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:28.860913 18832 consensus_queue.cc:237] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:27:28.877805 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:28.881561 20760 raft_consensus.cc:1203] T b15d11209e45473e863766e5cd6e822f P a14c220ad2cc4bf9bd7dad65f4803eb0 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240502 14:27:28.885988 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:28.915093 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:28.917233 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:28.922111 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:28.942476 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:28.954998 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:28.971539 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:28.986470 20894 raft_consensus.cc:1261] T 414e03d597934da3a6a74d786aeab134 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:28.988003 28743 consensus_queue.cc:1035] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:29.013257 28632 raft_consensus.cc:2932] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:29.029450 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:29.027463 20894 raft_consensus.cc:2932] T 414e03d597934da3a6a74d786aeab134 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:29.036476 18556 catalog_manager.cc:5484] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index 5 to 6, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:29.039295 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:29.163933 18832 consensus_queue.cc:237] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:27:29.165602 18945 tablet_service.cc:1514] Processing DeleteTablet for tablet 414e03d597934da3a6a74d786aeab134 with delete_type TABLET_DATA_TOMBSTONED (TS c902805060df4df9b5599d0e94b2dc06 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55752
I20240502 14:27:29.166806 20894 raft_consensus.cc:1261] T 414e03d597934da3a6a74d786aeab134 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 14:27:29.167480 18766 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 414e03d597934da3a6a74d786aeab134. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:27:29.167840 28743 consensus_queue.cc:1035] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:29.170188 28743 raft_consensus.cc:2932] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:29.175431 20894 raft_consensus.cc:2932] T 414e03d597934da3a6a74d786aeab134 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:29.182592 18543 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 414e03d597934da3a6a74d786aeab134 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:27:29.187541 28774 tablet_replica.cc:330] T 414e03d597934da3a6a74d786aeab134 P c902805060df4df9b5599d0e94b2dc06: stopping tablet replica
I20240502 14:27:29.190105 20740 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:29.187738 28774 raft_consensus.cc:2229] T 414e03d597934da3a6a74d786aeab134 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:29.191568 28774 raft_consensus.cc:2258] T 414e03d597934da3a6a74d786aeab134 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:29.195144 18832 consensus_queue.cc:237] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:27:29.199388 20626 raft_consensus.cc:1261] T 9580f3fdf7084cc0923b1b16bfedd450 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:27:29.199919 27390 consensus_queue.cc:1035] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:27:29.200829 18556 catalog_manager.cc:5484] T 414e03d597934da3a6a74d786aeab134 P 7ab6d395b30d4afb9821b61c9454430b reported cstate change: config changed from index 6 to 7, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) added. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } }
I20240502 14:27:29.201807 20894 raft_consensus.cc:1261] T 9580f3fdf7084cc0923b1b16bfedd450 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:27:29.202330 27390 consensus_queue.cc:1035] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:27:29.205691 21028 raft_consensus.cc:1261] T 9580f3fdf7084cc0923b1b16bfedd450 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:27:29.206429 27390 consensus_queue.cc:1035] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:27:29.220806 27390 raft_consensus.cc:2932] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:29.221617 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:29.222790 21028 raft_consensus.cc:2932] T 9580f3fdf7084cc0923b1b16bfedd450 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:29.197005 28774 ts_tablet_manager.cc:1875] T 414e03d597934da3a6a74d786aeab134 P c902805060df4df9b5599d0e94b2dc06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:29.237820 20626 raft_consensus.cc:2932] T 9580f3fdf7084cc0923b1b16bfedd450 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:29.243439 20894 raft_consensus.cc:2932] T 9580f3fdf7084cc0923b1b16bfedd450 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:29.249779 18553 catalog_manager.cc:5484] T 9580f3fdf7084cc0923b1b16bfedd450 P a834edffbcb2469aae598b069aab0681 reported cstate change: config changed from index 7 to 8, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) evicted. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:29.254397 18945 tablet_service.cc:1514] Processing DeleteTablet for tablet 9580f3fdf7084cc0923b1b16bfedd450 with delete_type TABLET_DATA_TOMBSTONED (TS c902805060df4df9b5599d0e94b2dc06 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:55752
I20240502 14:27:29.263311 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:29.265434 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:29.276345 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:29.285355 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:29.287142 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:29.318603 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:29.353724 28774 ts_tablet_manager.cc:1888] T 414e03d597934da3a6a74d786aeab134 P c902805060df4df9b5599d0e94b2dc06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:27:29.353852 28774 log.cc:1196] T 414e03d597934da3a6a74d786aeab134 P c902805060df4df9b5599d0e94b2dc06: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal/wals/414e03d597934da3a6a74d786aeab134
I20240502 14:27:29.368783 28774 tablet_replica.cc:330] T 9580f3fdf7084cc0923b1b16bfedd450 P c902805060df4df9b5599d0e94b2dc06: stopping tablet replica
I20240502 14:27:29.368839 18546 catalog_manager.cc:4830] TS c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409): tablet 414e03d597934da3a6a74d786aeab134 (table rebalance_test_table_4 [id=4dd18a179f134d1babc05959227a25cc]) successfully deleted
I20240502 14:27:29.369009 28774 raft_consensus.cc:2229] T 9580f3fdf7084cc0923b1b16bfedd450 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:29.369248 28774 raft_consensus.cc:2258] T 9580f3fdf7084cc0923b1b16bfedd450 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:29.369904 28774 ts_tablet_manager.cc:1875] T 9580f3fdf7084cc0923b1b16bfedd450 P c902805060df4df9b5599d0e94b2dc06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:29.407320 28774 ts_tablet_manager.cc:1888] T 9580f3fdf7084cc0923b1b16bfedd450 P c902805060df4df9b5599d0e94b2dc06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 14:27:29.407430 28774 log.cc:1196] T 9580f3fdf7084cc0923b1b16bfedd450 P c902805060df4df9b5599d0e94b2dc06: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-2/wal/wals/9580f3fdf7084cc0923b1b16bfedd450
I20240502 14:27:29.408079 18546 catalog_manager.cc:4830] TS c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409): tablet 9580f3fdf7084cc0923b1b16bfedd450 (table rebalance_test_table_0 [id=27066b01c29c4a1380ee4753a1c036d3]) successfully deleted
I20240502 14:27:29.565703 28841 ts_tablet_manager.cc:902] T 414e03d597934da3a6a74d786aeab134 P a14c220ad2cc4bf9bd7dad65f4803eb0: Initiating tablet copy from peer fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
F20240502 14:27:29.566133 28841 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0ffb6e582ef04fb38dd273c13febac58.queue_time_us, tablet-download-pool-0ffb6e582ef04fb38dd273c13febac58.queue_time_us) (tablet-download-pool-0ffb6e582ef04fb38dd273c13febac58.run_cpu_time_us, tablet-download-pool-0ffb6e582ef04fb38dd273c13febac58.run_cpu_time_us) (tablet-download-pool-0ffb6e582ef04fb38dd273c13febac58.run_wall_time_us, tablet-download-pool-0ffb6e582ef04fb38dd273c13febac58.run_wall_time_us) (tablet-download-pool-21af917743444f2db932636a78f33512.queue_time_us, tablet-download-pool-21af917743444f2db932636a78f33512.queue_time_us) (tablet-download-pool-21af917743444f2db932636a78f33512.run_cpu_time_us, tablet-download-pool-21af917743444f2db932636a78f33512.run_cpu_time_us) (tablet-download-pool-21af917743444f2db932636a78f33512.run_wall_time_us, tablet-download-pool-21af917743444f2db932636a78f33512.run_wall_time_us) (tablet-download-pool-246a8425400f4890a5264c4ae8a627c5.queue_time_us, tablet-download-pool-246a8425400f4890a5264c4ae8a627c5.queue_time_us) (tablet-download-pool-246a8425400f4890a5264c4ae8a627c5.run_cpu_time_us, tablet-download-pool-246a8425400f4890a5264c4ae8a627c5.run_cpu_time_us) (tablet-download-pool-246a8425400f4890a5264c4ae8a627c5.run_wall_time_us, tablet-download-pool-246a8425400f4890a5264c4ae8a627c5.run_wall_time_us) (tablet-download-pool-414e03d597934da3a6a74d786aeab134.queue_time_us, tablet-download-pool-414e03d597934da3a6a74d786aeab134.queue_time_us) (tablet-download-pool-522014370f0a4cc3bd31aa00d6a05c5e.queue_time_us, tablet-download-pool-522014370f0a4cc3bd31aa00d6a05c5e.queue_time_us) (tablet-download-pool-522014370f0a4cc3bd31aa00d6a05c5e.run_cpu_time_us, tablet-download-pool-522014370f0a4cc3bd31aa00d6a05c5e.run_cpu_time_us) (tablet-download-pool-522014370f0a4cc3bd31aa00d6a05c5e.run_wall_time_us, tablet-download-pool-522014370f0a4cc3bd31aa00d6a05c5e.run_wall_time_us) (tablet-download-pool-5be0d6f4f1874c79b906c4bdc34b0ce9.queue_time_us, tablet-download-pool-5be0d6f4f1874c79b906c4bdc34b0ce9.queue_time_us) (tablet-download-pool-5be0d6f4f1874c79b906c4bdc34b0ce9.run_cpu_time_us, tablet-download-pool-5be0d6f4f1874c79b906c4bdc34b0ce9.run_cpu_time_us) (tablet-download-pool-5be0d6f4f1874c79b906c4bdc34b0ce9.run_wall_time_us, tablet-download-pool-5be0d6f4f1874c79b906c4bdc34b0ce9.run_wall_time_us) (tablet-download-pool-67827c0262724897a05bf80df579bbf3.queue_time_us, tablet-download-pool-67827c0262724897a05bf80df579bbf3.queue_time_us) (tablet-download-pool-67827c0262724897a05bf80df579bbf3.run_cpu_time_us, tablet-download-pool-67827c0262724897a05bf80df579bbf3.run_cpu_time_us) (tablet-download-pool-67827c0262724897a05bf80df579bbf3.run_wall_time_us, tablet-download-pool-67827c0262724897a05bf80df579bbf3.run_wall_time_us) (tablet-download-pool-6a992d12aee7416096d1353533397386.queue_time_us, tablet-download-pool-6a992d12aee7416096d1353533397386.queue_time_us) (tablet-download-pool-6a992d12aee7416096d1353533397386.run_cpu_time_us, tablet-download-pool-6a992d12aee7416096d1353533397386.run_cpu_time_us) (tablet-download-pool-6a992d12aee7416096d1353533397386.run_wall_time_us, tablet-download-pool-6a992d12aee7416096d1353533397386.run_wall_time_us) (tablet-download-pool-6acde0e5509340fbb52bca18645a77b7.queue_time_us, tablet-download-pool-6acde0e5509340fbb52bca18645a77b7.queue_time_us) (tablet-download-pool-6acde0e5509340fbb52bca18645a77b7.run_cpu_time_us, tablet-download-pool-6acde0e5509340fbb52bca18645a77b7.run_cpu_time_us) (tablet-download-pool-6acde0e5509340fbb52bca18645a77b7.run_wall_time_us, tablet-download-pool-6acde0e5509340fbb52bca18645a77b7.run_wall_time_us) (tablet-download-pool-96662a3d9e1b4f03b05464ddd06b93d6.queue_time_us, tablet-download-pool-96662a3d9e1b4f03b05464ddd06b93d6.queue_time_us) (tablet-download-pool-96662a3d9e1b4f03b05464ddd06b93d6.run_cpu_time_us, tablet-download-pool-96662a3d9e1b4f03b05464ddd06b93d6.run_cpu_time_us) (tablet-download-pool-96662a3d9e1b4f03b05464ddd06b93d6.run_wall_time_us, tablet-download-pool-96662a3d9e1b4f03b05464ddd06b93d6.run_wall_time_us) (tablet-download-pool-b15d11209e45473e863766e5cd6e822f.queue_time_us, tablet-download-pool-b15d11209e45473e863766e5cd6e822f.queue_time_us) (tablet-download-pool-b15d11209e45473e863766e5cd6e822f.run_cpu_time_us, tablet-download-pool-b15d11209e45473e863766e5cd6e822f.run_cpu_time_us) (tablet-download-pool-b15d11209e45473e863766e5cd6e822f.run_wall_time_us, tablet-download-pool-b15d11209e45473e863766e5cd6e822f.run_wall_time_us) (tablet-download-pool-b315327141954b53a0f87c2d1dc14b9b.queue_time_us, tablet-download-pool-b315327141954b53a0f87c2d1dc14b9b.queue_time_us) (tablet-download-pool-b315327141954b53a0f87c2d1dc14b9b.run_cpu_time_us, tablet-download-pool-b315327141954b53a0f87c2d1dc14b9b.run_cpu_time_us) (tablet-download-pool-b315327141954b53a0f87c2d1dc14b9b.run_wall_time_us, tablet-download-pool-b315327141954b53a0f87c2d1dc14b9b.run_wall_time_us) (tablet-download-pool-bad03a4ec8494938839d7d6eef8c684e.queue_time_us, tablet-download-pool-bad03a4ec8494938839d7d6eef8c684e.queue_time_us) (tablet-download-pool-bad03a4ec8494938839d7d6eef8c684e.run_cpu_time_us, tablet-download-pool-bad03a4ec8494938839d7d6eef8c684e.run_cpu_time_us) (tablet-download-pool-bad03a4ec8494938839d7d6eef8c684e.run_wall_time_us, tablet-download-pool-bad03a4ec8494938839d7d6eef8c684e.run_wall_time_us) (tablet-download-pool-cf2a6426ae6c438f942d39530da6e3e3.queue_time_us, tablet-download-pool-cf2a6426ae6c438f942d39530da6e3e3.queue_time_us) (tablet-download-pool-cf2a6426ae6c438f942d39530da6e3e3.run_cpu_time_us, tablet-download-pool-cf2a6426ae6c438f942d39530da6e3e3.run_cpu_time_us) (tablet-download-pool-cf2a6426ae6c438f942d39530da6e3e3.run_wall_time_us, tablet-download-pool-cf2a6426ae6c438f942d39530da6e3e3.run_wall_time_us) (tablet-download-pool-e5fb3b049ba94af998791640bd0e31c5.queue_time_us, tablet-download-pool-e5fb3b049ba94af998791640bd0e31c5.queue_time_us) (tablet-download-pool-e5fb3b049ba94af998791640bd0e31c5.run_cpu_time_us, tablet-download-pool-e5fb3b049ba94af998791640bd0e31c5.run_cpu_time_us) (tablet-download-pool-e5fb3b049ba94af998791640bd0e31c5.run_wall_time_us, tablet-download-pool-e5fb3b049ba94af998791640bd0e31c5.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240502 14:27:29.569041 21028 consensus_queue.cc:237] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
@ 0x7faaeaddadcd google::LogMessage::Fail() at ??:0
@ 0x7faaeaddeb93 google::LogMessage::SendToLog() at ??:0
@ 0x7faaeadda7cc google::LogMessage::Flush() at ??:0
@ 0x7faaeaddbf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 14:27:29.580956 18699 raft_consensus.cc:1261] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Refusing update from remote peer ff6acba046404ef0a6f41ad5e09d852f: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:27:29.581581 27781 consensus_queue.cc:1035] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:27:29.632472 18699 raft_consensus.cc:2932] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:29.630394 28854 raft_consensus.cc:2932] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
@ 0x7faaedb1b742 kudu::TraceMetrics::InternName() at ??:0
I20240502 14:27:29.637630 18553 catalog_manager.cc:5484] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b reported cstate change: config changed from index 6 to 7, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New cstate: current_term: 2 leader_uuid: "ff6acba046404ef0a6f41ad5e09d852f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:29.641866 19076 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ecc6539b85d40e49df7ffc5f93cb801 with delete_type TABLET_DATA_TOMBSTONED (TS 007b2e79dc424a6f80141180f072f2d8 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:41468
I20240502 14:27:29.656059 28861 tablet_replica.cc:330] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8: stopping tablet replica
I20240502 14:27:29.657909 28861 raft_consensus.cc:2229] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:29.658109 28861 raft_consensus.cc:2258] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:29.658807 28861 ts_tablet_manager.cc:1875] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:29.680487 21028 consensus_queue.cc:237] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:27:29.683283 18699 raft_consensus.cc:1261] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Refusing update from remote peer ff6acba046404ef0a6f41ad5e09d852f: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240502 14:27:29.683954 20964 consensus_peers.cc:485] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463): Couldn't send request to peer 007b2e79dc424a6f80141180f072f2d8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:27:29.684746 28854 consensus_queue.cc:1035] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:27:29.688270 27781 raft_consensus.cc:2932] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
@ 0x7faaedae7f8b kudu::ThreadPool::ThreadPool() at ??:0
I20240502 14:27:29.693929 18699 raft_consensus.cc:2932] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:27:29.700379 18553 catalog_manager.cc:5484] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b reported cstate change: config changed from index 7 to 8, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) added. New cstate: current_term: 2 leader_uuid: "ff6acba046404ef0a6f41ad5e09d852f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:27:29.708717 18963 consensus_queue.cc:237] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
@ 0x7faaedae1cd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240502 14:27:29.714267 28861 ts_tablet_manager.cc:1888] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 14:27:29.714396 28861 log.cc:1196] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal/wals/1ecc6539b85d40e49df7ffc5f93cb801
I20240502 14:27:29.715188 18546 catalog_manager.cc:4830] TS 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463): tablet 1ecc6539b85d40e49df7ffc5f93cb801 (table rebalance_test_table_9 [id=a9acd50a500c4f84a41c8d9ba23ae431]) successfully deleted
I20240502 14:27:29.722558 18545 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 1ecc6539b85d40e49df7ffc5f93cb801 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
@ 0x7faafcf348de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7faafcf4cdaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7faafd077480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7faafd0afa8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7faafd092981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7faafd09270e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7faaedaebb75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7faaedaef19c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
I20240502 14:27:29.776418 20625 raft_consensus.cc:1261] T ecb8a9b04ba34f779e2227e77e6f764c P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:27:29.777001 28720 consensus_queue.cc:1035] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
@ 0x7faaedaef019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7faaedaca01e kudu::Thread::SuperviseThread() at ??:0
@ 0x7faaf0e4f6db start_thread at ??:0
I20240502 14:27:29.789052 20894 raft_consensus.cc:1261] T ecb8a9b04ba34f779e2227e77e6f764c P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:27:29.789746 28720 consensus_queue.cc:1035] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
@ 0x7faae9c8d71f clone at ??:0
I20240502 14:27:29.803308 20625 raft_consensus.cc:2932] T ecb8a9b04ba34f779e2227e77e6f764c P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:29.804574 20892 raft_consensus.cc:2932] T ecb8a9b04ba34f779e2227e77e6f764c P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:29.801596 28905 raft_consensus.cc:2932] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:29.812242 18553 catalog_manager.cc:5484] T ecb8a9b04ba34f779e2227e77e6f764c P a834edffbcb2469aae598b069aab0681 reported cstate change: config changed from index 6 to 7, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:29.815814 18681 tablet_service.cc:1514] Processing DeleteTablet for tablet ecb8a9b04ba34f779e2227e77e6f764c with delete_type TABLET_DATA_TOMBSTONED (TS 2db304b538e14857b7b1cc272d0db94b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54454
I20240502 14:27:29.844609 28909 tablet_replica.cc:330] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b: stopping tablet replica
I20240502 14:27:29.844789 28909 raft_consensus.cc:2229] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:29.845093 28909 raft_consensus.cc:2258] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:29.845804 28909 ts_tablet_manager.cc:1875] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:29.930119 28909 ts_tablet_manager.cc:1888] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:27:29.930260 28909 log.cc:1196] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal/wals/ecb8a9b04ba34f779e2227e77e6f764c
I20240502 14:27:29.931037 18544 catalog_manager.cc:4830] TS 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749): tablet ecb8a9b04ba34f779e2227e77e6f764c (table rebalance_test_table_3 [id=210510c054044948932a30db816646aa]) successfully deleted
I20240502 14:27:29.953723 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:29.993477 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:30.043505 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:30.051764 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:30.054698 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:30.071218 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:30.123800 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:30.126294 18699 consensus_queue.cc:237] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:27:30.194640 20892 raft_consensus.cc:1261] T d28db16c9f7e405b8866f0e6c548e246 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:27:30.194769 20625 raft_consensus.cc:1261] T d28db16c9f7e405b8866f0e6c548e246 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 14:27:30.195365 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463): Couldn't send request to peer 007b2e79dc424a6f80141180f072f2d8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:27:30.195628 28443 consensus_queue.cc:1035] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:27:30.197037 28443 consensus_queue.cc:1035] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:27:30.219290 28983 ts_tablet_manager.cc:902] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8: Initiating tablet copy from peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871)
I20240502 14:27:30.228914 28443 raft_consensus.cc:2932] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:27:30.234340 20625 raft_consensus.cc:2932] T d28db16c9f7e405b8866f0e6c548e246 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:27:30.245574 20892 raft_consensus.cc:2932] T d28db16c9f7e405b8866f0e6c548e246 P 7ab6d395b30d4afb9821b61c9454430b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:27:30.246667 18554 catalog_manager.cc:5484] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b reported cstate change: config changed from index 7 to 8, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) added. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:27:30.277204 28983 tablet_copy_client.cc:308] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.72:35871
I20240502 14:27:30.278195 21049 tablet_copy_service.cc:140] P ff6acba046404ef0a6f41ad5e09d852f: Received BeginTabletCopySession request for tablet 1ecc6539b85d40e49df7ffc5f93cb801 from peer 007b2e79dc424a6f80141180f072f2d8 ({username='slave'} at 127.5.107.68:40869)
I20240502 14:27:30.278427 21049 tablet_copy_service.cc:161] P ff6acba046404ef0a6f41ad5e09d852f: Beginning new tablet copy session on tablet 1ecc6539b85d40e49df7ffc5f93cb801 from peer 007b2e79dc424a6f80141180f072f2d8 at {username='slave'} at 127.5.107.68:40869: session id = 007b2e79dc424a6f80141180f072f2d8-1ecc6539b85d40e49df7ffc5f93cb801
I20240502 14:27:30.283545 21049 tablet_copy_source_session.cc:205] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:27:30.286406 28983 ts_tablet_manager.cc:1875] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:27:30.296643 28983 ts_tablet_manager.cc:1888] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240502 14:27:30.297011 28983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ecc6539b85d40e49df7ffc5f93cb801. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:27:30.309268 28983 tablet_copy_client.cc:783] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8: tablet copy: Starting download of 0 data blocks...
I20240502 14:27:30.309489 28983 tablet_copy_client.cc:647] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8: tablet copy: Starting download of 1 WAL segments...
I20240502 14:27:30.311614 28983 tablet_copy_client.cc:523] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:27:30.366895 28983 tablet_bootstrap.cc:492] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8: Bootstrap starting.
W20240502 14:27:30.384985 18898 connection.cc:535] client connection to 127.5.107.70:46021 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:27:30.385289 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:27:30.402665 18635 connection.cc:535] client connection to 127.5.107.70:46021 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:27:30.402906 19030 connection.cc:535] client connection to 127.5.107.70:46021 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:27:30.402948 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:27:30.403081 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:27:30.403676 20560 connection.cc:535] client connection to 127.5.107.70:46021 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:27:30.403846 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:27:30.404475 18766 connection.cc:535] client connection to 127.5.107.70:46021 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:27:30.404675 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:27:30.431993 29031 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:27:30.404787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.107.70:46021 (local address 127.5.107.68:44137)
0502 14:27:30.422086 (+ 17299us) negotiation.cc:107] Waiting for socket to connect
0502 14:27:30.422103 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0502 14:27:30.422251 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:27:30.430965 (+ 8714us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":17209,"thread_start_us":17564,"threads_started":1}
W20240502 14:27:30.432215 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:27:30.432323 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:27:30.432401 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:27:30.457249 29034 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:27:30.414529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.107.70:46021 (local address 127.5.107.66:39467)
0502 14:27:30.456110 (+ 41581us) negotiation.cc:107] Waiting for socket to connect
0502 14:27:30.456257 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":41453,"thread_start_us":41377,"threads_started":1}
W20240502 14:27:30.457406 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:27:30.457588 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection reset by peer (error 104)
W20240502 14:27:30.457671 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:27:30.457738 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:27:30.461882 18898 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.70:46021, user_credentials={real_user=slave}} blocked reactor thread for 76660.3us
W20240502 14:27:30.469472 29033 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:27:30.405334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.107.70:46021 (local address 127.5.107.65:34533)
0502 14:27:30.468303 (+ 62969us) negotiation.cc:107] Waiting for socket to connect
0502 14:27:30.468457 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":62757,"threads_started":1}
W20240502 14:27:30.470656 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:27:30.470777 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:27:30.470850 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:27:30.470835 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:27:30.471982 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:27:30.472097 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:27:30.472175 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:27:30.477237 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:27:30.514910 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:27:30.550024 28983 tablet_bootstrap.cc:492] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:27:30.550487 28983 tablet_bootstrap.cc:492] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8: Bootstrap complete.
I20240502 14:27:30.550971 28983 ts_tablet_manager.cc:1367] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8: Time spent bootstrapping tablet: real 0.184s user 0.059s sys 0.024s
I20240502 14:27:30.552939 28983 raft_consensus.cc:348] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:27:30.553141 28983 raft_consensus.cc:729] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Initialized, Role: FOLLOWER
I20240502 14:27:30.553465 28983 consensus_queue.cc:260] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:27:30.556511 28983 ts_tablet_manager.cc:1398] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:27:30.557405 21049 tablet_copy_service.cc:342] P ff6acba046404ef0a6f41ad5e09d852f: Request end of tablet copy session 007b2e79dc424a6f80141180f072f2d8-1ecc6539b85d40e49df7ffc5f93cb801 received from {username='slave'} at 127.5.107.68:40869
I20240502 14:27:30.557548 21049 tablet_copy_service.cc:433] P ff6acba046404ef0a6f41ad5e09d852f: ending tablet copy session 007b2e79dc424a6f80141180f072f2d8-1ecc6539b85d40e49df7ffc5f93cb801 on tablet 1ecc6539b85d40e49df7ffc5f93cb801 with peer 007b2e79dc424a6f80141180f072f2d8
I20240502 14:27:30.575650 19093 consensus_queue.cc:237] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:27:30.605888 20626 raft_consensus.cc:1261] T 6a992d12aee7416096d1353533397386 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 14:27:30.607048 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749): Couldn't send request to peer 2db304b538e14857b7b1cc272d0db94b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:27:30.607307 28848 consensus_queue.cc:1035] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 14:27:30.615327 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:27:30.709553 19093 raft_consensus.cc:1203] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240502 14:27:30.712538 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:30.758019 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:30.779769 28983 ts_tablet_manager.cc:902] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8: Initiating tablet copy from peer 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:27:30.785401 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:30.803961 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:30.823997 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:30.833623 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:30.860496 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:30.909886 28983 tablet_copy_client.cc:308] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.65:36749
I20240502 14:27:30.910642 18721 tablet_copy_service.cc:140] P 2db304b538e14857b7b1cc272d0db94b: Received BeginTabletCopySession request for tablet d28db16c9f7e405b8866f0e6c548e246 from peer 007b2e79dc424a6f80141180f072f2d8 ({username='slave'} at 127.5.107.68:35397)
I20240502 14:27:30.910789 18721 tablet_copy_service.cc:161] P 2db304b538e14857b7b1cc272d0db94b: Beginning new tablet copy session on tablet d28db16c9f7e405b8866f0e6c548e246 from peer 007b2e79dc424a6f80141180f072f2d8 at {username='slave'} at 127.5.107.68:35397: session id = 007b2e79dc424a6f80141180f072f2d8-d28db16c9f7e405b8866f0e6c548e246
I20240502 14:27:30.913851 18721 tablet_copy_source_session.cc:205] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:27:30.915335 28983 ts_tablet_manager.cc:1875] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:27:30.928133 28983 ts_tablet_manager.cc:1888] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
W20240502 14:27:30.928891 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:27:30.941473 28983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d28db16c9f7e405b8866f0e6c548e246. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:27:30.953058 28983 tablet_copy_client.cc:783] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8: tablet copy: Starting download of 0 data blocks...
I20240502 14:27:30.953351 28983 tablet_copy_client.cc:647] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8: tablet copy: Starting download of 1 WAL segments...
I20240502 14:27:30.956202 28983 tablet_copy_client.cc:523] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:27:30.964200 28983 tablet_bootstrap.cc:492] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8: Bootstrap starting.
I20240502 14:27:31.079864 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:31.096488 28983 tablet_bootstrap.cc:492] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:27:31.096930 28983 tablet_bootstrap.cc:492] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8: Bootstrap complete.
I20240502 14:27:31.097205 28983 ts_tablet_manager.cc:1367] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8: Time spent bootstrapping tablet: real 0.133s user 0.057s sys 0.033s
I20240502 14:27:31.097944 28983 raft_consensus.cc:348] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:27:31.098196 28983 raft_consensus.cc:729] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Initialized, Role: FOLLOWER
I20240502 14:27:31.098459 28983 consensus_queue.cc:260] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:27:31.099048 28983 ts_tablet_manager.cc:1398] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:27:31.155108 18721 tablet_copy_service.cc:342] P 2db304b538e14857b7b1cc272d0db94b: Request end of tablet copy session 007b2e79dc424a6f80141180f072f2d8-d28db16c9f7e405b8866f0e6c548e246 received from {username='slave'} at 127.5.107.68:35397
I20240502 14:27:31.155269 18721 tablet_copy_service.cc:433] P 2db304b538e14857b7b1cc272d0db94b: ending tablet copy session 007b2e79dc424a6f80141180f072f2d8-d28db16c9f7e405b8866f0e6c548e246 on tablet d28db16c9f7e405b8866f0e6c548e246 with peer 007b2e79dc424a6f80141180f072f2d8
I20240502 14:27:31.177527 29176 ts_tablet_manager.cc:902] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b: Initiating tablet copy from peer 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:27:31.212266 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:31.225533 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:31.249311 19093 raft_consensus.cc:1203] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 14:27:31.271481 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:31.275223 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:31.287793 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:31.289392 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:31.294351 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:31.299103 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:31.306732 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:31.309875 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:31.325240 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:31.364110 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:31.383667 29176 tablet_copy_client.cc:308] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.68:41463
I20240502 14:27:31.384819 19117 tablet_copy_service.cc:140] P 007b2e79dc424a6f80141180f072f2d8: Received BeginTabletCopySession request for tablet 6a992d12aee7416096d1353533397386 from peer 2db304b538e14857b7b1cc272d0db94b ({username='slave'} at 127.5.107.65:60769)
I20240502 14:27:31.384954 19117 tablet_copy_service.cc:161] P 007b2e79dc424a6f80141180f072f2d8: Beginning new tablet copy session on tablet 6a992d12aee7416096d1353533397386 from peer 2db304b538e14857b7b1cc272d0db94b at {username='slave'} at 127.5.107.65:60769: session id = 2db304b538e14857b7b1cc272d0db94b-6a992d12aee7416096d1353533397386
I20240502 14:27:31.387625 19117 tablet_copy_source_session.cc:205] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:27:31.390761 29176 ts_tablet_manager.cc:1875] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b: Deleting tablet data with delete state TABLET_DATA_COPYING
W20240502 14:27:31.410046 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:27:31.411836 29176 ts_tablet_manager.cc:1888] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240502 14:27:31.412072 29176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a992d12aee7416096d1353533397386. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:27:31.423779 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:31.454401 29176 tablet_copy_client.cc:783] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b: tablet copy: Starting download of 0 data blocks...
I20240502 14:27:31.454778 29176 tablet_copy_client.cc:647] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:27:31.457101 29176 tablet_copy_client.cc:523] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:27:31.470049 29176 tablet_bootstrap.cc:492] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b: Bootstrap starting.
W20240502 14:27:31.561512 18768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.72:35871, user_credentials={real_user=slave}} blocked reactor thread for 69179.1us
I20240502 14:27:31.666152 29176 tablet_bootstrap.cc:492] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:27:31.666795 29176 tablet_bootstrap.cc:492] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b: Bootstrap complete.
I20240502 14:27:31.667110 29176 ts_tablet_manager.cc:1367] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b: Time spent bootstrapping tablet: real 0.197s user 0.062s sys 0.000s
I20240502 14:27:31.667976 29176 raft_consensus.cc:348] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:27:31.668496 29176 raft_consensus.cc:729] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Initialized, Role: FOLLOWER
I20240502 14:27:31.668735 29176 consensus_queue.cc:260] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:27:31.669517 29176 ts_tablet_manager.cc:1398] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 14:27:31.674117 19117 tablet_copy_service.cc:342] P 007b2e79dc424a6f80141180f072f2d8: Request end of tablet copy session 2db304b538e14857b7b1cc272d0db94b-6a992d12aee7416096d1353533397386 received from {username='slave'} at 127.5.107.65:60769
I20240502 14:27:31.674304 19117 tablet_copy_service.cc:433] P 007b2e79dc424a6f80141180f072f2d8: ending tablet copy session 2db304b538e14857b7b1cc272d0db94b-6a992d12aee7416096d1353533397386 on tablet 6a992d12aee7416096d1353533397386 with peer 2db304b538e14857b7b1cc272d0db94b
I20240502 14:27:31.706871 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:31.714357 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:31.716688 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:31.731340 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:31.758038 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:31.763743 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:31.805300 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:32.004920 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:27:32.091223 21081 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 14:26:43.157902 21086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:26:43.172933 21086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:26:43.206573 21086 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20240502 14:26:44.011705 21213 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.5.107.126:33551, user_credentials={real_user=slave}} blocked reactor thread for 48593.2us
I20240502 14:26:46.635111 21086 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 14:26:48.095499 21639 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.Quiesce -> {remote=127.5.107.71:39195, user_credentials={real_user=slave}} blocked reactor thread for 43535.1us
W20240502 14:26:48.655997 21086 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5a8a0451cdf0479c94edf1d94e77fb83: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server a14c220ad2cc4bf9bd7dad65f4803eb0
W20240502 14:26:48.656315 21086 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 995bc008b9f94332bfa2c42a77e41bb2: could not find any suitable replica to move from server 2db304b538e14857b7b1cc272d0db94b to server a834edffbcb2469aae598b069aab0681
W20240502 14:26:48.656435 21086 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff5cff7e35c746a5bc64cc06f18bf7d9: could not find any suitable replica to move from server 007b2e79dc424a6f80141180f072f2d8 to server a14c220ad2cc4bf9bd7dad65f4803eb0
W20240502 14:26:48.656725 21086 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 210510c054044948932a30db816646aa: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server 7ab6d395b30d4afb9821b61c9454430b
W20240502 14:26:48.656833 21086 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 27066b01c29c4a1380ee4753a1c036d3: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server a14c220ad2cc4bf9bd7dad65f4803eb0
W20240502 14:26:48.657032 21086 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b260896c8b8744c0a1b4f691850aec93: could not find any suitable replica to move from server 2db304b538e14857b7b1cc272d0db94b to server a14c220ad2cc4bf9bd7dad65f4803eb0
W20240502 14:26:48.657145 21086 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3670a8bef9694c029d2128006cc913d3: could not find any suitable replica to move from server fec04c50dc5b49878ffa3c3873df32e4 to server ff6acba046404ef0a6f41ad5e09d852f
W20240502 14:26:48.657409 21086 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0e84c82d60e64dbd9149c5f9fb7ff3c8: could not find any suitable replica to move from server 2db304b538e14857b7b1cc272d0db94b to server a14c220ad2cc4bf9bd7dad65f4803eb0
W20240502 14:26:48.657519 21086 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a9acd50a500c4f84a41c8d9ba23ae431: could not find any suitable replica to move from server 007b2e79dc424a6f80141180f072f2d8 to server ff6acba046404ef0a6f41ad5e09d852f
W20240502 14:26:48.657722 21086 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4dd18a179f134d1babc05959227a25cc: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server 7ab6d395b30d4afb9821b61c9454430b
W20240502 14:26:48.657928 21086 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 995bc008b9f94332bfa2c42a77e41bb2: could not find any suitable replica to move from server fec04c50dc5b49878ffa3c3873df32e4 to server ff6acba046404ef0a6f41ad5e09d852f
W20240502 14:26:48.658075 21086 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff5cff7e35c746a5bc64cc06f18bf7d9: could not find any suitable replica to move from server 2db304b538e14857b7b1cc272d0db94b to server 7ab6d395b30d4afb9821b61c9454430b
W20240502 14:26:48.658195 21086 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3670a8bef9694c029d2128006cc913d3: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server a14c220ad2cc4bf9bd7dad65f4803eb0
W20240502 14:26:48.658329 21086 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0e84c82d60e64dbd9149c5f9fb7ff3c8: could not find any suitable replica to move from server 007b2e79dc424a6f80141180f072f2d8 to server a834edffbcb2469aae598b069aab0681
I20240502 14:26:49.411759 21086 rebalancer_tool.cc:1074] tablet 332e4b23751b422ab7bd17b80c93701d: 'c902805060df4df9b5599d0e94b2dc06' -> 'ff6acba046404ef0a6f41ad5e09d852f' move scheduled
I20240502 14:26:49.666443 21086 rebalancer_tool.cc:1074] tablet 9580f3fdf7084cc0923b1b16bfedd450: '2db304b538e14857b7b1cc272d0db94b' -> '7ab6d395b30d4afb9821b61c9454430b' move scheduled
I20240502 14:26:50.752692 21086 rebalancer_tool.cc:1074] tablet 8937c809610542d88fdde3daf002c890: 'fec04c50dc5b49878ffa3c3873df32e4' -> 'a834edffbcb2469aae598b069aab0681' move scheduled
I20240502 14:26:51.207885 21086 rebalancer_tool.cc:1074] tablet b315327141954b53a0f87c2d1dc14b9b: '007b2e79dc424a6f80141180f072f2d8' -> 'a14c220ad2cc4bf9bd7dad65f4803eb0' move scheduled
I20240502 14:26:51.636729 21086 rebalancer_tool.cc:1074] tablet 8b8d81c275144f898dcc69a6d98d11e0: 'c902805060df4df9b5599d0e94b2dc06' -> '7ab6d395b30d4afb9821b61c9454430b' move scheduled
I20240502 14:26:52.235709 21086 rebalancer_tool.cc:1074] tablet 0d8667d91fd64809b635362c4a8fa33f: '2db304b538e14857b7b1cc272d0db94b' -> 'ff6acba046404ef0a6f41ad5e09d852f' move scheduled
I20240502 14:26:52.778744 21086 rebalancer_tool.cc:1074] tablet 522014370f0a4cc3bd31aa00d6a05c5e: 'fec04c50dc5b49878ffa3c3873df32e4' -> 'a834edffbcb2469aae598b069aab0681' move scheduled
I20240502 14:26:53.424858 21086 rebalancer_tool.cc:1074] tablet 96662a3d9e1b4f03b05464ddd06b93d6: '007b2e79dc424a6f80141180f072f2d8' -> '7ab6d395b30d4afb9821b61c9454430b' move scheduled
I20240502 14:26:54.023172 21086 rebalancer_tool.cc:1074] tablet 0ffb6e582ef04fb38dd273c13febac58: 'c902805060df4df9b5599d0e94b2dc06' -> 'a14c220ad2cc4bf9bd7dad65f4803eb0' move scheduled
I20240502 14:26:54.616261 21086 rebalancer_tool.cc:1074] tablet b15d11209e45473e863766e5cd6e822f: '2db304b538e14857b7b1cc272d0db94b' -> 'ff6acba046404ef0a6f41ad5e09d852f' move scheduled
I20240502 14:26:55.103670 21086 rebalancer_tool.cc:1084] tablet b13bcd6e693f4c7687a1aabdafbb83c6: 'fec04c50dc5b49878ffa3c3873df32e4' -> 'a14c220ad2cc4bf9bd7dad65f4803eb0' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240502 14:26:55.103842 21086 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:26:57.091315 21086 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff5cff7e35c746a5bc64cc06f18bf7d9: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server a834edffbcb2469aae598b069aab0681
W20240502 14:26:57.091490 21086 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a9acd50a500c4f84a41c8d9ba23ae431: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server a14c220ad2cc4bf9bd7dad65f4803eb0
W20240502 14:26:57.091766 21086 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff5cff7e35c746a5bc64cc06f18bf7d9: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server 7ab6d395b30d4afb9821b61c9454430b
W20240502 14:26:57.091976 21086 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff5cff7e35c746a5bc64cc06f18bf7d9: could not find any suitable replica to move from server 2db304b538e14857b7b1cc272d0db94b to server a14c220ad2cc4bf9bd7dad65f4803eb0
W20240502 14:26:57.092137 21086 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0e84c82d60e64dbd9149c5f9fb7ff3c8: could not find any suitable replica to move from server 2db304b538e14857b7b1cc272d0db94b to server a14c220ad2cc4bf9bd7dad65f4803eb0
W20240502 14:26:57.092362 21086 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4dd18a179f134d1babc05959227a25cc: could not find any suitable replica to move from server fec04c50dc5b49878ffa3c3873df32e4 to server 7ab6d395b30d4afb9821b61c9454430b
W20240502 14:26:57.092463 21086 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a9acd50a500c4f84a41c8d9ba23ae431: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server a14c220ad2cc4bf9bd7dad65f4803eb0
W20240502 14:26:57.092584 21086 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5a8a0451cdf0479c94edf1d94e77fb83: could not find any suitable replica to move from server 2db304b538e14857b7b1cc272d0db94b to server a834edffbcb2469aae598b069aab0681
I20240502 14:26:57.226289 21086 rebalancer_tool.cc:1074] tablet 21af917743444f2db932636a78f33512: 'fec04c50dc5b49878ffa3c3873df32e4' -> '' move scheduled
I20240502 14:26:57.378870 21086 rebalancer_tool.cc:1074] tablet cf2a6426ae6c438f942d39530da6e3e3: '007b2e79dc424a6f80141180f072f2d8' -> '' move scheduled
I20240502 14:26:57.869475 21086 rebalancer_tool.cc:1074] tablet ecb8a9b04ba34f779e2227e77e6f764c: '2db304b538e14857b7b1cc272d0db94b' -> 'a834edffbcb2469aae598b069aab0681' move scheduled
I20240502 14:26:57.946795 21086 rebalancer_tool.cc:1074] tablet 6acde0e5509340fbb52bca18645a77b7: 'c902805060df4df9b5599d0e94b2dc06' -> '' move scheduled
I20240502 14:26:58.719935 21086 rebalancer_tool.cc:1074] tablet db56136f2dbb429098d4d78b15a3a439: '007b2e79dc424a6f80141180f072f2d8' -> 'ff6acba046404ef0a6f41ad5e09d852f' move scheduled
I20240502 14:26:59.206557 21086 rebalancer_tool.cc:1074] tablet 246a8425400f4890a5264c4ae8a627c5: '007b2e79dc424a6f80141180f072f2d8' -> '7ab6d395b30d4afb9821b61c9454430b' move scheduled
I20240502 14:26:59.704900 21086 rebalancer_tool.cc:1074] tablet 51ef0be93c914e258635e13f3cca79aa: 'fec04c50dc5b49878ffa3c3873df32e4' -> 'a834edffbcb2469aae598b069aab0681' move scheduled
I20240502 14:26:59.745847 21086 rebalancer_tool.cc:1074] tablet b13bcd6e693f4c7687a1aabdafbb83c6: '2db304b538e14857b7b1cc272d0db94b' -> '' move scheduled
I20240502 14:26:59.798971 21086 rebalancer_tool.cc:1074] tablet df5f14206a1d46e4866f34fa234f945b: 'c902805060df4df9b5599d0e94b2dc06' -> '' move scheduled
I20240502 14:27:00.139865 21086 rebalancer_tool.cc:1074] tablet d28db16c9f7e405b8866f0e6c548e246: 'fec04c50dc5b49878ffa3c3873df32e4' -> 'a834edffbcb2469aae598b069aab0681' move scheduled
I20240502 14:27:11.910182 21086 rebalancer_tool.cc:1136] tablet b13bcd6e693f4c7687a1aabdafbb83c6: '2db304b538e14857b7b1cc272d0db94b' -> '' move completed: OK
I20240502 14:27:13.504217 21086 rebalancer_tool.cc:1123] tablet 332e4b23751b422ab7bd17b80c93701d: 'c902805060df4df9b5599d0e94b2dc06' -> 'ff6acba046404ef0a6f41ad5e09d852f' move is abandoned: Illegal state: Leader has not yet committed an operation in its own term
I20240502 14:27:13.670739 21086 rebalancer_tool.cc:1136] tablet cf2a6426ae6c438f942d39530da6e3e3: '007b2e79dc424a6f80141180f072f2d8' -> '' move completed: OK
I20240502 14:27:13.784440 21086 rebalancer_tool.cc:1136] tablet 8937c809610542d88fdde3daf002c890: 'fec04c50dc5b49878ffa3c3873df32e4' -> 'a834edffbcb2469aae598b069aab0681' move completed: OK
I20240502 14:27:13.934798 21086 rebalancer_tool.cc:1136] tablet 9580f3fdf7084cc0923b1b16bfedd450: '2db304b538e14857b7b1cc272d0db94b' -> '7ab6d395b30d4afb9821b61c9454430b' move completed: OK
I20240502 14:27:17.827312 21086 rebalancer_tool.cc:1136] tablet 0ffb6e582ef04fb38dd273c13febac58: 'c902805060df4df9b5599d0e94b2dc06' -> 'a14c220ad2cc4bf9bd7dad65f4803eb0' move completed: OK
W20240502 14:27:21.184494 27074 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.5.107.68:41463, user_credentials={real_user=slave}} blocked reactor thread for 82465.8us
I20240502 14:27:23.965243 21086 rebalancer_tool.cc:1136] tablet 21af917743444f2db932636a78f33512: 'fec04c50dc5b49878ffa3c3873df32e4' -> '' move completed: OK
I20240502 14:27:29.874119 21086 rebalancer_tool.cc:1136] tablet 6acde0e5509340fbb52bca18645a77b7: 'c902805060df4df9b5599d0e94b2dc06' -> '' move completed: OK
I20240502 14:27:29.908751 21086 rebalancer_tool.cc:1074] tablet 097eb705c9e24740a5be61bf3354f3d8: 'fec04c50dc5b49878ffa3c3873df32e4' -> '' move scheduled
I20240502 14:27:30.075875 21086 rebalancer_tool.cc:1074] tablet bad03a4ec8494938839d7d6eef8c684e: '007b2e79dc424a6f80141180f072f2d8' -> '' move scheduled
I20240502 14:27:30.197501 21086 rebalancer_tool.cc:1084] tablet 1ecc6539b85d40e49df7ffc5f93cb801: '2db304b538e14857b7b1cc272d0db94b' -> 'ff6acba046404ef0a6f41ad5e09d852f' move ignored: Illegal state: tablet 1ecc6539b85d40e49df7ffc5f93cb801: replica ff6acba046404ef0a6f41ad5e09d852f is already present
I20240502 14:27:30.197676 21086 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): unacceptable health status UNAVAILABLE
I20240502 14:27:32.226097 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:32.242697 18699 raft_consensus.cc:1203] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 14:27:32.258675 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:32.261900 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:32.265039 29344 raft_consensus.cc:2932] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:27:32.275722 29344 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:51978 (request call id 0) took 1701 ms. Trace:
I20240502 14:27:32.276152 29344 rpcz_store.cc:276] 0502 14:27:30.574657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:27:30.574746 (+ 89us) service_pool.cc:225] Handling call
0502 14:27:30.576750 (+ 2004us) log.cc:842] Serialized 482 byte log entry
0502 14:27:32.275695 (+1698945us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4991,"raft.run_cpu_time_us":598,"raft.run_wall_time_us":2446,"thread_start_us":33081,"threads_started":2,"wal-append.queue_time_us":23662,"wal-append.run_cpu_time_us":581,"wal-append.run_wall_time_us":1000998}
I20240502 14:27:32.282975 18553 catalog_manager.cc:5484] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: config changed from index 7 to 8, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) added. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:27:32.300312 20626 raft_consensus.cc:2932] T 6a992d12aee7416096d1353533397386 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
W20240502 14:27:32.365417 18900 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.71:39195, user_credentials={real_user=slave}} blocked reactor thread for 117966us
I20240502 14:27:32.376585 18699 raft_consensus.cc:2932] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:27:32.392447 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:32.399854 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:32.413779 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:32.418876 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:32.424206 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:32.465539 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:32.487835 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:32.495882 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:27:32.496098 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:27:32.496151 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:32.518050 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:27:32.530092 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:32.543671 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:32.547869 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:27:32.566603 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:27:32.567618 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:27:32.585943 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:32.646689 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:27:32.678561 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:27:32.716269 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:27:32.744164 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:27:32.765465 18699 consensus_queue.cc:237] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:27:32.768911 21028 raft_consensus.cc:1261] T db56136f2dbb429098d4d78b15a3a439 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 14:27:32.768934 18634 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009): Couldn't send request to peer fec04c50dc5b49878ffa3c3873df32e4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:27:32.769552 29306 consensus_queue.cc:1035] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:27:32.779563 20626 raft_consensus.cc:1261] T db56136f2dbb429098d4d78b15a3a439 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:27:32.780599 29408 consensus_queue.cc:1035] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:27:32.802927 29306 raft_consensus.cc:2932] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } }
I20240502 14:27:32.805590 21028 raft_consensus.cc:2932] T db56136f2dbb429098d4d78b15a3a439 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } }
I20240502 14:27:32.816634 20626 raft_consensus.cc:2932] T db56136f2dbb429098d4d78b15a3a439 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } }
I20240502 14:27:32.823081 18556 catalog_manager.cc:5484] T db56136f2dbb429098d4d78b15a3a439 P a834edffbcb2469aae598b069aab0681 reported cstate change: config changed from index 7 to 8, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) added. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } }
W20240502 14:27:32.882490 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:27:32.897815 21028 consensus_queue.cc:237] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
W20240502 14:27:32.903687 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:27:32.908914 18699 raft_consensus.cc:1261] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Refusing update from remote peer ff6acba046404ef0a6f41ad5e09d852f: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:27:32.909667 28881 consensus_queue.cc:1035] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 14:27:32.927109 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:27:32.941311 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:32.952220 29450 raft_consensus.cc:2932] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:32.953927 18699 raft_consensus.cc:2932] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:32.961505 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:32.964347 18553 catalog_manager.cc:5484] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b reported cstate change: config changed from index 8 to 9, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) evicted. New cstate: current_term: 2 leader_uuid: "ff6acba046404ef0a6f41ad5e09d852f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:32.974689 21028 consensus_queue.cc:237] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:27:32.985625 18699 raft_consensus.cc:1261] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Refusing update from remote peer ff6acba046404ef0a6f41ad5e09d852f: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:27:32.986935 28881 consensus_queue.cc:1035] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:27:32.990650 28881 raft_consensus.cc:2932] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:32.994319 18701 raft_consensus.cc:2932] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:33.001559 18545 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 1ecc6539b85d40e49df7ffc5f93cb801 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:27:33.005060 18560 catalog_manager.cc:5484] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f reported cstate change: config changed from index 9 to 10, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) added. New cstate: current_term: 2 leader_uuid: "ff6acba046404ef0a6f41ad5e09d852f" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:27:33.014242 19076 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ecc6539b85d40e49df7ffc5f93cb801 with delete_type TABLET_DATA_TOMBSTONED (TS 007b2e79dc424a6f80141180f072f2d8 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:41468
W20240502 14:27:33.059573 20964 consensus_peers.cc:485] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ecc6539b85d40e49df7ffc5f93cb801. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:27:33.074326 29463 tablet_replica.cc:330] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8: stopping tablet replica
I20240502 14:27:33.078187 29463 raft_consensus.cc:2229] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:33.080050 29463 raft_consensus.cc:2258] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:33.081267 29463 ts_tablet_manager.cc:1875] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:33.103035 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:33.127106 18701 consensus_queue.cc:237] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:27:33.127791 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:33.136210 29463 ts_tablet_manager.cc:1888] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240502 14:27:33.136355 29463 log.cc:1196] T 1ecc6539b85d40e49df7ffc5f93cb801 P 007b2e79dc424a6f80141180f072f2d8: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-3/wal/wals/1ecc6539b85d40e49df7ffc5f93cb801
I20240502 14:27:33.137228 18546 catalog_manager.cc:4830] TS 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463): tablet 1ecc6539b85d40e49df7ffc5f93cb801 (table rebalance_test_table_9 [id=a9acd50a500c4f84a41c8d9ba23ae431]) successfully deleted
W20240502 14:27:33.141782 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:27:33.162074 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:27:33.163583 18769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.71:39195, user_credentials={real_user=slave}} blocked reactor thread for 48226.2us
I20240502 14:27:33.163681 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:33.174857 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:27:33.179973 29491 ts_tablet_manager.cc:902] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4: Initiating tablet copy from peer 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:27:33.187399 21028 raft_consensus.cc:1261] T 0ffb6e582ef04fb38dd273c13febac58 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:27:33.188763 29417 consensus_queue.cc:1035] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 14:27:33.189328 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:27:33.201807 18637 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409): Couldn't send request to peer c902805060df4df9b5599d0e94b2dc06. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:27:33.210212 29491 tablet_copy_client.cc:308] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.65:36749
I20240502 14:27:33.211133 18721 tablet_copy_service.cc:140] P 2db304b538e14857b7b1cc272d0db94b: Received BeginTabletCopySession request for tablet db56136f2dbb429098d4d78b15a3a439 from peer fec04c50dc5b49878ffa3c3873df32e4 ({username='slave'} at 127.5.107.66:57219)
I20240502 14:27:33.211287 18721 tablet_copy_service.cc:161] P 2db304b538e14857b7b1cc272d0db94b: Beginning new tablet copy session on tablet db56136f2dbb429098d4d78b15a3a439 from peer fec04c50dc5b49878ffa3c3873df32e4 at {username='slave'} at 127.5.107.66:57219: session id = fec04c50dc5b49878ffa3c3873df32e4-db56136f2dbb429098d4d78b15a3a439
I20240502 14:27:33.214025 18721 tablet_copy_source_session.cc:205] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:27:33.217043 29491 ts_tablet_manager.cc:1875] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:27:33.218808 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:33.229481 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:33.250389 29491 ts_tablet_manager.cc:1888] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 14:27:33.250742 29491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db56136f2dbb429098d4d78b15a3a439. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:27:33.263700 29491 tablet_copy_client.cc:783] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4: tablet copy: Starting download of 0 data blocks...
I20240502 14:27:33.264181 29491 tablet_copy_client.cc:647] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4: tablet copy: Starting download of 1 WAL segments...
I20240502 14:27:33.270314 29491 tablet_copy_client.cc:523] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:27:33.277735 29491 tablet_bootstrap.cc:492] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4: Bootstrap starting.
I20240502 14:27:33.373138 29491 tablet_bootstrap.cc:492] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:27:33.373693 29491 tablet_bootstrap.cc:492] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4: Bootstrap complete.
I20240502 14:27:33.374269 29491 ts_tablet_manager.cc:1367] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4: Time spent bootstrapping tablet: real 0.097s user 0.061s sys 0.003s
I20240502 14:27:33.375743 29491 raft_consensus.cc:348] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:27:33.376031 29491 raft_consensus.cc:729] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Initialized, Role: FOLLOWER
I20240502 14:27:33.376292 29491 consensus_queue.cc:260] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:27:33.377117 29491 ts_tablet_manager.cc:1398] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:27:33.390717 18721 tablet_copy_service.cc:342] P 2db304b538e14857b7b1cc272d0db94b: Request end of tablet copy session fec04c50dc5b49878ffa3c3873df32e4-db56136f2dbb429098d4d78b15a3a439 received from {username='slave'} at 127.5.107.66:57219
I20240502 14:27:33.390882 18721 tablet_copy_service.cc:433] P 2db304b538e14857b7b1cc272d0db94b: ending tablet copy session fec04c50dc5b49878ffa3c3873df32e4-db56136f2dbb429098d4d78b15a3a439 on tablet db56136f2dbb429098d4d78b15a3a439 with peer fec04c50dc5b49878ffa3c3873df32e4
I20240502 14:27:33.401568 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:33.436877 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:33.437420 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:33.445148 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:33.453159 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:33.458789 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:33.521149 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:33.530577 29547 ts_tablet_manager.cc:902] T 1ecc6539b85d40e49df7ffc5f93cb801 P 7ab6d395b30d4afb9821b61c9454430b: Initiating tablet copy from peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871)
F20240502 14:27:33.530954 29547 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0d8667d91fd64809b635362c4a8fa33f.queue_time_us, tablet-download-pool-0d8667d91fd64809b635362c4a8fa33f.queue_time_us) (tablet-download-pool-0d8667d91fd64809b635362c4a8fa33f.run_cpu_time_us, tablet-download-pool-0d8667d91fd64809b635362c4a8fa33f.run_cpu_time_us) (tablet-download-pool-0d8667d91fd64809b635362c4a8fa33f.run_wall_time_us, tablet-download-pool-0d8667d91fd64809b635362c4a8fa33f.run_wall_time_us) (tablet-download-pool-1ecc6539b85d40e49df7ffc5f93cb801.queue_time_us, tablet-download-pool-1ecc6539b85d40e49df7ffc5f93cb801.queue_time_us) (tablet-download-pool-246a8425400f4890a5264c4ae8a627c5.queue_time_us, tablet-download-pool-246a8425400f4890a5264c4ae8a627c5.queue_time_us) (tablet-download-pool-246a8425400f4890a5264c4ae8a627c5.run_cpu_time_us, tablet-download-pool-246a8425400f4890a5264c4ae8a627c5.run_cpu_time_us) (tablet-download-pool-246a8425400f4890a5264c4ae8a627c5.run_wall_time_us, tablet-download-pool-246a8425400f4890a5264c4ae8a627c5.run_wall_time_us) (tablet-download-pool-414e03d597934da3a6a74d786aeab134.queue_time_us, tablet-download-pool-414e03d597934da3a6a74d786aeab134.queue_time_us) (tablet-download-pool-414e03d597934da3a6a74d786aeab134.run_cpu_time_us, tablet-download-pool-414e03d597934da3a6a74d786aeab134.run_cpu_time_us) (tablet-download-pool-414e03d597934da3a6a74d786aeab134.run_wall_time_us, tablet-download-pool-414e03d597934da3a6a74d786aeab134.run_wall_time_us) (tablet-download-pool-51ef0be93c914e258635e13f3cca79aa.queue_time_us, tablet-download-pool-51ef0be93c914e258635e13f3cca79aa.queue_time_us) (tablet-download-pool-51ef0be93c914e258635e13f3cca79aa.run_cpu_time_us, tablet-download-pool-51ef0be93c914e258635e13f3cca79aa.run_cpu_time_us) (tablet-download-pool-51ef0be93c914e258635e13f3cca79aa.run_wall_time_us, tablet-download-pool-51ef0be93c914e258635e13f3cca79aa.run_wall_time_us) (tablet-download-pool-67827c0262724897a05bf80df579bbf3.queue_time_us, tablet-download-pool-67827c0262724897a05bf80df579bbf3.queue_time_us) (tablet-download-pool-67827c0262724897a05bf80df579bbf3.run_cpu_time_us, tablet-download-pool-67827c0262724897a05bf80df579bbf3.run_cpu_time_us) (tablet-download-pool-67827c0262724897a05bf80df579bbf3.run_wall_time_us, tablet-download-pool-67827c0262724897a05bf80df579bbf3.run_wall_time_us) (tablet-download-pool-8937c809610542d88fdde3daf002c890.queue_time_us, tablet-download-pool-8937c809610542d88fdde3daf002c890.queue_time_us) (tablet-download-pool-8937c809610542d88fdde3daf002c890.run_cpu_time_us, tablet-download-pool-8937c809610542d88fdde3daf002c890.run_cpu_time_us) (tablet-download-pool-8937c809610542d88fdde3daf002c890.run_wall_time_us, tablet-download-pool-8937c809610542d88fdde3daf002c890.run_wall_time_us) (tablet-download-pool-8b8d81c275144f898dcc69a6d98d11e0.queue_time_us, tablet-download-pool-8b8d81c275144f898dcc69a6d98d11e0.queue_time_us) (tablet-download-pool-8b8d81c275144f898dcc69a6d98d11e0.run_cpu_time_us, tablet-download-pool-8b8d81c275144f898dcc69a6d98d11e0.run_cpu_time_us) (tablet-download-pool-8b8d81c275144f898dcc69a6d98d11e0.run_wall_time_us, tablet-download-pool-8b8d81c275144f898dcc69a6d98d11e0.run_wall_time_us) (tablet-download-pool-9580f3fdf7084cc0923b1b16bfedd450.queue_time_us, tablet-download-pool-9580f3fdf7084cc0923b1b16bfedd450.queue_time_us) (tablet-download-pool-9580f3fdf7084cc0923b1b16bfedd450.run_cpu_time_us, tablet-download-pool-9580f3fdf7084cc0923b1b16bfedd450.run_cpu_time_us) (tablet-download-pool-9580f3fdf7084cc0923b1b16bfedd450.run_wall_time_us, tablet-download-pool-9580f3fdf7084cc0923b1b16bfedd450.run_wall_time_us) (tablet-download-pool-96662a3d9e1b4f03b05464ddd06b93d6.queue_time_us, tablet-download-pool-96662a3d9e1b4f03b05464ddd06b93d6.queue_time_us) (tablet-download-pool-96662a3d9e1b4f03b05464ddd06b93d6.run_cpu_time_us, tablet-download-pool-96662a3d9e1b4f03b05464ddd06b93d6.run_cpu_time_us) (tablet-download-pool-96662a3d9e1b4f03b05464ddd06b93d6.run_wall_time_us, tablet-download-pool-96662a3d9e1b4f03b05464ddd06b93d6.run_wall_time_us) (tablet-download-pool-b13bcd6e693f4c7687a1aabdafbb83c6.queue_time_us, tablet-download-pool-b13bcd6e693f4c7687a1aabdafbb83c6.queue_time_us) (tablet-download-pool-b13bcd6e693f4c7687a1aabdafbb83c6.run_cpu_time_us, tablet-download-pool-b13bcd6e693f4c7687a1aabdafbb83c6.run_cpu_time_us) (tablet-download-pool-b13bcd6e693f4c7687a1aabdafbb83c6.run_wall_time_us, tablet-download-pool-b13bcd6e693f4c7687a1aabdafbb83c6.run_wall_time_us) (tablet-download-pool-b315327141954b53a0f87c2d1dc14b9b.queue_time_us, tablet-download-pool-b315327141954b53a0f87c2d1dc14b9b.queue_time_us) (tablet-download-pool-b315327141954b53a0f87c2d1dc14b9b.run_cpu_time_us, tablet-download-pool-b315327141954b53a0f87c2d1dc14b9b.run_cpu_time_us) (tablet-download-pool-b315327141954b53a0f87c2d1dc14b9b.run_wall_time_us, tablet-download-pool-b315327141954b53a0f87c2d1dc14b9b.run_wall_time_us) (tablet-download-pool-cf2a6426ae6c438f942d39530da6e3e3.queue_time_us, tablet-download-pool-cf2a6426ae6c438f942d39530da6e3e3.queue_time_us) (tablet-download-pool-cf2a6426ae6c438f942d39530da6e3e3.run_cpu_time_us, tablet-download-pool-cf2a6426ae6c438f942d39530da6e3e3.run_cpu_time_us) (tablet-download-pool-cf2a6426ae6c438f942d39530da6e3e3.run_wall_time_us, tablet-download-pool-cf2a6426ae6c438f942d39530da6e3e3.run_wall_time_us) (tablet-download-pool-d28db16c9f7e405b8866f0e6c548e246.queue_time_us, tablet-download-pool-d28db16c9f7e405b8866f0e6c548e246.queue_time_us) (tablet-download-pool-d28db16c9f7e405b8866f0e6c548e246.run_cpu_time_us, tablet-download-pool-d28db16c9f7e405b8866f0e6c548e246.run_cpu_time_us) (tablet-download-pool-d28db16c9f7e405b8866f0e6c548e246.run_wall_time_us, tablet-download-pool-d28db16c9f7e405b8866f0e6c548e246.run_wall_time_us) (tablet-download-pool-ecb8a9b04ba34f779e2227e77e6f764c.queue_time_us, tablet-download-pool-ecb8a9b04ba34f779e2227e77e6f764c.queue_time_us) (tablet-download-pool-ecb8a9b04ba34f779e2227e77e6f764c.run_cpu_time_us, tablet-download-pool-ecb8a9b04ba34f779e2227e77e6f764c.run_cpu_time_us) (tablet-download-pool-ecb8a9b04ba34f779e2227e77e6f764c.run_wall_time_us, tablet-download-pool-ecb8a9b04ba34f779e2227e77e6f764c.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240502 14:27:33.534185 18832 raft_consensus.cc:1203] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240502 14:27:33.543206 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
@ 0x7feaff30ddcd google::LogMessage::Fail() at ??:0
@ 0x7feaff311b93 google::LogMessage::SendToLog() at ??:0
@ 0x7feaff30d7cc google::LogMessage::Flush() at ??:0
@ 0x7feaff30ef59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 14:27:33.599615 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:33.600689 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:33.647967 20872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:27:33.682371 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:33.694615 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
@ 0x7feb0204e742 kudu::TraceMetrics::InternName() at ??:0
I20240502 14:27:33.717590 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:33.740921 29568 ts_tablet_manager.cc:902] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06: Initiating tablet copy from peer 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
@ 0x7feb0201af8b kudu::ThreadPool::ThreadPool() at ??:0
I20240502 14:27:33.755503 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
@ 0x7feb02014cd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240502 14:27:33.810174 29568 tablet_copy_client.cc:308] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.65:36749
I20240502 14:27:33.811620 18721 tablet_copy_service.cc:140] P 2db304b538e14857b7b1cc272d0db94b: Received BeginTabletCopySession request for tablet 0ffb6e582ef04fb38dd273c13febac58 from peer c902805060df4df9b5599d0e94b2dc06 ({username='slave'} at 127.5.107.67:55959)
I20240502 14:27:33.811764 18721 tablet_copy_service.cc:161] P 2db304b538e14857b7b1cc272d0db94b: Beginning new tablet copy session on tablet 0ffb6e582ef04fb38dd273c13febac58 from peer c902805060df4df9b5599d0e94b2dc06 at {username='slave'} at 127.5.107.67:55959: session id = c902805060df4df9b5599d0e94b2dc06-0ffb6e582ef04fb38dd273c13febac58
I20240502 14:27:33.815132 18721 tablet_copy_source_session.cc:205] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:27:33.816834 29568 ts_tablet_manager.cc:1875] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:27:33.828516 29568 ts_tablet_manager.cc:1888] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240502 14:27:33.828856 29568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ffb6e582ef04fb38dd273c13febac58. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7feb114678de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 14:27:33.843169 29568 tablet_copy_client.cc:783] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06: tablet copy: Starting download of 0 data blocks...
I20240502 14:27:33.845238 29568 tablet_copy_client.cc:647] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06: tablet copy: Starting download of 1 WAL segments...
I20240502 14:27:33.851465 29568 tablet_copy_client.cc:523] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:27:33.856007 29568 tablet_bootstrap.cc:492] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06: Bootstrap starting.
@ 0x7feb1147fdaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7feb115aa480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240502 14:27:33.925801 29568 tablet_bootstrap.cc:492] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:27:33.926371 29568 tablet_bootstrap.cc:492] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06: Bootstrap complete.
I20240502 14:27:33.926656 29568 ts_tablet_manager.cc:1367] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06: Time spent bootstrapping tablet: real 0.071s user 0.052s sys 0.010s
I20240502 14:27:33.927435 29568 raft_consensus.cc:348] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } }
I20240502 14:27:33.928254 29568 raft_consensus.cc:729] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Initialized, Role: FOLLOWER
I20240502 14:27:33.928483 29568 consensus_queue.cc:260] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:27:33.932291 29568 ts_tablet_manager.cc:1398] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 14:27:33.933619 18721 tablet_copy_service.cc:342] P 2db304b538e14857b7b1cc272d0db94b: Request end of tablet copy session c902805060df4df9b5599d0e94b2dc06-0ffb6e582ef04fb38dd273c13febac58 received from {username='slave'} at 127.5.107.67:55959
I20240502 14:27:33.933730 18721 tablet_copy_service.cc:433] P 2db304b538e14857b7b1cc272d0db94b: ending tablet copy session c902805060df4df9b5599d0e94b2dc06-0ffb6e582ef04fb38dd273c13febac58 on tablet 0ffb6e582ef04fb38dd273c13febac58 with peer c902805060df4df9b5599d0e94b2dc06
@ 0x7feb115e2a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7feb115c5981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7feb115c570e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7feb0201eb75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7feb0202219c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7feb02022019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7feb01ffd01e kudu::Thread::SuperviseThread() at ??:0
@ 0x7feb053826db start_thread at ??:0
@ 0x7feafe1c071f clone at ??:0
W20240502 14:27:34.059952 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:27:34.122859 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:34.138216 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:34.142506 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:34.150274 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:34.153762 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:34.196581 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:34.206842 18963 raft_consensus.cc:1203] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 14:27:34.208985 29417 raft_consensus.cc:2932] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
I20240502 14:27:34.218818 18963 raft_consensus.cc:2932] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
I20240502 14:27:34.225440 29417 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:57052 (request call id 0) took 1099 ms. Trace:
I20240502 14:27:34.225596 29417 rpcz_store.cc:276] 0502 14:27:33.126275 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 14:27:33.126336 (+ 61us) service_pool.cc:225] Handling call
0502 14:27:33.128403 (+ 2067us) log.cc:842] Serialized 482 byte log entry
0502 14:27:34.225401 (+1096998us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":89,"raft.queue_time_us":486,"raft.run_cpu_time_us":625,"raft.run_wall_time_us":626,"thread_start_us":38286,"threads_started":1,"wal-append.queue_time_us":37508,"wal-append.run_cpu_time_us":587,"wal-append.run_wall_time_us":1000636}
I20240502 14:27:34.227442 21028 raft_consensus.cc:2932] T 0ffb6e582ef04fb38dd273c13febac58 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
I20240502 14:27:34.233618 18556 catalog_manager.cc:5484] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b reported cstate change: config changed from index 7 to 8, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) added. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a14c220ad2cc4bf9bd7dad65f4803eb0" member_type: VOTER last_known_addr { host: "127.5.107.70" port: 46021 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
I20240502 14:27:34.453416 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:34.484927 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:34.556826 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:34.564644 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:34.582468 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:34.586704 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:34.667140 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:27:34.678714 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:27:34.688500 19032 connection.cc:535] client connection to 127.5.107.71:39195 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:27:34.688963 20562 connection.cc:535] client connection to 127.5.107.71:39195 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:27:34.690654 18637 connection.cc:535] client connection to 127.5.107.71:39195 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:27:34.691304 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.107.71:39195 (error 108)
W20240502 14:27:34.703140 18900 connection.cc:535] client connection to 127.5.107.71:39195 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:27:34.703517 18769 connection.cc:535] client connection to 127.5.107.71:39195 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:27:34.705503 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:27:34.705605 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:27:34.705662 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:27:34.705720 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:27:34.705741 29031 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:27:34.689369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.107.71:39195 (local address 127.5.107.68:42595)
0502 14:27:34.689658 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0502 14:27:34.689679 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0502 14:27:34.689796 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:27:34.705598 (+ 15802us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":158}
W20240502 14:27:34.705787 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:27:34.706770 29376 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:27:34.689803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.107.71:39195 (local address 127.5.107.69:42259)
0502 14:27:34.690082 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0502 14:27:34.690099 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0502 14:27:34.690234 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:27:34.706108 (+ 15874us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":139}
W20240502 14:27:34.706974 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:27:34.709619 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:27:34.711861 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:27:34.711975 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:27:34.712045 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:27:34.712298 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:27:34.712424 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:27:34.712512 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:27:34.712594 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:27:34.734099 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:27:34.760535 18701 consensus_queue.cc:237] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:27:34.780390 20626 raft_consensus.cc:1261] T df5f14206a1d46e4866f34fa234f945b P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 14:27:34.781124 18637 consensus_peers.cc:485] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b -> Peer c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409): Couldn't send request to peer c902805060df4df9b5599d0e94b2dc06. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:27:34.781082 21080 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 14:26:42.719754 21085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:26:42.720256 21085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:26:42.733589 21085 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:26:45.547391 21085 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 14:26:47.631750 21085 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a9acd50a500c4f84a41c8d9ba23ae431: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server ff6acba046404ef0a6f41ad5e09d852f
W20240502 14:26:47.631958 21085 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 27066b01c29c4a1380ee4753a1c036d3: could not find any suitable replica to move from server fec04c50dc5b49878ffa3c3873df32e4 to server 7ab6d395b30d4afb9821b61c9454430b
W20240502 14:26:47.632062 21085 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4dd18a179f134d1babc05959227a25cc: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server a14c220ad2cc4bf9bd7dad65f4803eb0
W20240502 14:26:47.632242 21085 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3670a8bef9694c029d2128006cc913d3: could not find any suitable replica to move from server 007b2e79dc424a6f80141180f072f2d8 to server a14c220ad2cc4bf9bd7dad65f4803eb0
W20240502 14:26:47.632349 21085 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 995bc008b9f94332bfa2c42a77e41bb2: could not find any suitable replica to move from server 2db304b538e14857b7b1cc272d0db94b to server a834edffbcb2469aae598b069aab0681
W20240502 14:26:47.632483 21085 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff5cff7e35c746a5bc64cc06f18bf7d9: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server a834edffbcb2469aae598b069aab0681
W20240502 14:26:47.632609 21085 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0e84c82d60e64dbd9149c5f9fb7ff3c8: could not find any suitable replica to move from server fec04c50dc5b49878ffa3c3873df32e4 to server a834edffbcb2469aae598b069aab0681
W20240502 14:26:47.632752 21085 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 27066b01c29c4a1380ee4753a1c036d3: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server a14c220ad2cc4bf9bd7dad65f4803eb0
W20240502 14:26:47.632856 21085 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 210510c054044948932a30db816646aa: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server ff6acba046404ef0a6f41ad5e09d852f
W20240502 14:26:47.632992 21085 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5a8a0451cdf0479c94edf1d94e77fb83: could not find any suitable replica to move from server 2db304b538e14857b7b1cc272d0db94b to server ff6acba046404ef0a6f41ad5e09d852f
W20240502 14:26:47.633082 21085 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b260896c8b8744c0a1b4f691850aec93: could not find any suitable replica to move from server fec04c50dc5b49878ffa3c3873df32e4 to server 7ab6d395b30d4afb9821b61c9454430b
W20240502 14:26:47.633239 21085 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3670a8bef9694c029d2128006cc913d3: could not find any suitable replica to move from server fec04c50dc5b49878ffa3c3873df32e4 to server a834edffbcb2469aae598b069aab0681
W20240502 14:26:47.633358 21085 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff5cff7e35c746a5bc64cc06f18bf7d9: could not find any suitable replica to move from server 007b2e79dc424a6f80141180f072f2d8 to server a14c220ad2cc4bf9bd7dad65f4803eb0
W20240502 14:26:47.633486 21085 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 995bc008b9f94332bfa2c42a77e41bb2: could not find any suitable replica to move from server fec04c50dc5b49878ffa3c3873df32e4 to server 7ab6d395b30d4afb9821b61c9454430b
W20240502 14:26:47.633587 21085 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0e84c82d60e64dbd9149c5f9fb7ff3c8: could not find any suitable replica to move from server 2db304b538e14857b7b1cc272d0db94b to server ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:26:48.302048 21085 rebalancer_tool.cc:1074] tablet 9580f3fdf7084cc0923b1b16bfedd450: '2db304b538e14857b7b1cc272d0db94b' -> 'a834edffbcb2469aae598b069aab0681' move scheduled
I20240502 14:26:49.170411 21085 rebalancer_tool.cc:1074] tablet 0ffb6e582ef04fb38dd273c13febac58: 'c902805060df4df9b5599d0e94b2dc06' -> 'ff6acba046404ef0a6f41ad5e09d852f' move scheduled
I20240502 14:26:49.499347 21085 rebalancer_tool.cc:1074] tablet e5fb3b049ba94af998791640bd0e31c5: 'fec04c50dc5b49878ffa3c3873df32e4' -> 'a14c220ad2cc4bf9bd7dad65f4803eb0' move scheduled
I20240502 14:26:49.996117 21085 rebalancer_tool.cc:1074] tablet 0d8667d91fd64809b635362c4a8fa33f: '007b2e79dc424a6f80141180f072f2d8' -> '7ab6d395b30d4afb9821b61c9454430b' move scheduled
I20240502 14:26:51.062458 21085 rebalancer_tool.cc:1074] tablet 522014370f0a4cc3bd31aa00d6a05c5e: '2db304b538e14857b7b1cc272d0db94b' -> 'ff6acba046404ef0a6f41ad5e09d852f' move scheduled
I20240502 14:26:51.461503 21085 rebalancer_tool.cc:1074] tablet df5f14206a1d46e4866f34fa234f945b: 'c902805060df4df9b5599d0e94b2dc06' -> 'a834edffbcb2469aae598b069aab0681' move scheduled
I20240502 14:26:52.267977 21085 rebalancer_tool.cc:1074] tablet 21af917743444f2db932636a78f33512: 'fec04c50dc5b49878ffa3c3873df32e4' -> 'a14c220ad2cc4bf9bd7dad65f4803eb0' move scheduled
I20240502 14:26:52.667389 21085 rebalancer_tool.cc:1074] tablet 414e03d597934da3a6a74d786aeab134: '007b2e79dc424a6f80141180f072f2d8' -> '7ab6d395b30d4afb9821b61c9454430b' move scheduled
I20240502 14:26:53.326848 21085 rebalancer_tool.cc:1074] tablet 8b8d81c275144f898dcc69a6d98d11e0: '2db304b538e14857b7b1cc272d0db94b' -> 'ff6acba046404ef0a6f41ad5e09d852f' move scheduled
I20240502 14:26:53.920331 21085 rebalancer_tool.cc:1074] tablet 2c2afaffde96462eacd0d32bbdec23b1: 'c902805060df4df9b5599d0e94b2dc06' -> 'a834edffbcb2469aae598b069aab0681' move scheduled
I20240502 14:26:54.464362 21085 rebalancer_tool.cc:1074] tablet 8937c809610542d88fdde3daf002c890: 'fec04c50dc5b49878ffa3c3873df32e4' -> '7ab6d395b30d4afb9821b61c9454430b' move scheduled
I20240502 14:26:54.943300 21085 rebalancer_tool.cc:1074] tablet 96662a3d9e1b4f03b05464ddd06b93d6: '007b2e79dc424a6f80141180f072f2d8' -> 'a14c220ad2cc4bf9bd7dad65f4803eb0' move scheduled
I20240502 14:26:55.109553 21085 rebalancer_tool.cc:1084] tablet b15d11209e45473e863766e5cd6e822f: '2db304b538e14857b7b1cc272d0db94b' -> 'ff6acba046404ef0a6f41ad5e09d852f' move ignored: Illegal state: tablet b15d11209e45473e863766e5cd6e822f: replica ff6acba046404ef0a6f41ad5e09d852f is already present
I20240502 14:26:55.109939 21085 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:26:56.885818 21085 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff5cff7e35c746a5bc64cc06f18bf7d9: could not find any suitable replica to move from server 2db304b538e14857b7b1cc272d0db94b to server a834edffbcb2469aae598b069aab0681
W20240502 14:26:56.886188 21085 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff5cff7e35c746a5bc64cc06f18bf7d9: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server a14c220ad2cc4bf9bd7dad65f4803eb0
W20240502 14:26:56.886324 21085 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5a8a0451cdf0479c94edf1d94e77fb83: could not find any suitable replica to move from server 007b2e79dc424a6f80141180f072f2d8 to server a834edffbcb2469aae598b069aab0681
W20240502 14:26:56.886427 21085 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a9acd50a500c4f84a41c8d9ba23ae431: could not find any suitable replica to move from server 2db304b538e14857b7b1cc272d0db94b to server 7ab6d395b30d4afb9821b61c9454430b
W20240502 14:26:56.895188 21085 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b260896c8b8744c0a1b4f691850aec93: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server a834edffbcb2469aae598b069aab0681
W20240502 14:26:56.895363 21085 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0e84c82d60e64dbd9149c5f9fb7ff3c8: could not find any suitable replica to move from server fec04c50dc5b49878ffa3c3873df32e4 to server a14c220ad2cc4bf9bd7dad65f4803eb0
W20240502 14:26:56.896644 21085 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff5cff7e35c746a5bc64cc06f18bf7d9: could not find any suitable replica to move from server 2db304b538e14857b7b1cc272d0db94b to server a14c220ad2cc4bf9bd7dad65f4803eb0
I20240502 14:26:57.055483 21085 rebalancer_tool.cc:1074] tablet b13bcd6e693f4c7687a1aabdafbb83c6: '2db304b538e14857b7b1cc272d0db94b' -> '' move scheduled
I20240502 14:26:57.127990 21085 rebalancer_tool.cc:1074] tablet bad03a4ec8494938839d7d6eef8c684e: 'c902805060df4df9b5599d0e94b2dc06' -> '' move scheduled
I20240502 14:26:57.260172 21085 rebalancer_tool.cc:1074] tablet 097eb705c9e24740a5be61bf3354f3d8: 'fec04c50dc5b49878ffa3c3873df32e4' -> '' move scheduled
I20240502 14:26:57.643323 21085 rebalancer_tool.cc:1074] tablet 246a8425400f4890a5264c4ae8a627c5: '2db304b538e14857b7b1cc272d0db94b' -> 'a14c220ad2cc4bf9bd7dad65f4803eb0' move scheduled
I20240502 14:26:57.813917 21085 rebalancer_tool.cc:1074] tablet cf2a6426ae6c438f942d39530da6e3e3: '007b2e79dc424a6f80141180f072f2d8' -> '' move scheduled
I20240502 14:26:58.057346 21085 rebalancer_tool.cc:1084] tablet 1ecc6539b85d40e49df7ffc5f93cb801: 'c902805060df4df9b5599d0e94b2dc06' -> 'ff6acba046404ef0a6f41ad5e09d852f' move ignored: Illegal state: tablet 1ecc6539b85d40e49df7ffc5f93cb801: replica ff6acba046404ef0a6f41ad5e09d852f is already present
I20240502 14:26:58.057507 21085 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:26:59.301676 23532 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.5.107.70:46021, user_credentials={real_user=slave}} blocked reactor thread for 74326.1us
W20240502 14:26:59.756310 21085 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 27066b01c29c4a1380ee4753a1c036d3: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server a14c220ad2cc4bf9bd7dad65f4803eb0
W20240502 14:26:59.760864 21085 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4dd18a179f134d1babc05959227a25cc: could not find any suitable replica to move from server fec04c50dc5b49878ffa3c3873df32e4 to server ff6acba046404ef0a6f41ad5e09d852f
W20240502 14:26:59.761117 21085 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4dd18a179f134d1babc05959227a25cc: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server a14c220ad2cc4bf9bd7dad65f4803eb0
W20240502 14:26:59.761237 21085 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3670a8bef9694c029d2128006cc913d3: could not find any suitable replica to move from server fec04c50dc5b49878ffa3c3873df32e4 to server 7ab6d395b30d4afb9821b61c9454430b
W20240502 14:26:59.761384 21085 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 210510c054044948932a30db816646aa: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server ff6acba046404ef0a6f41ad5e09d852f
W20240502 14:26:59.761520 21085 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0e84c82d60e64dbd9149c5f9fb7ff3c8: could not find any suitable replica to move from server fec04c50dc5b49878ffa3c3873df32e4 to server a834edffbcb2469aae598b069aab0681
W20240502 14:26:59.761631 21085 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b260896c8b8744c0a1b4f691850aec93: could not find any suitable replica to move from server 007b2e79dc424a6f80141180f072f2d8 to server a14c220ad2cc4bf9bd7dad65f4803eb0
W20240502 14:26:59.761741 21085 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 27066b01c29c4a1380ee4753a1c036d3: could not find any suitable replica to move from server 2db304b538e14857b7b1cc272d0db94b to server a14c220ad2cc4bf9bd7dad65f4803eb0
I20240502 14:26:59.926617 21085 rebalancer_tool.cc:1074] tablet 1ecc6539b85d40e49df7ffc5f93cb801: '007b2e79dc424a6f80141180f072f2d8' -> '' move scheduled
I20240502 14:27:10.202566 21085 rebalancer_tool.cc:1136] tablet 21af917743444f2db932636a78f33512: 'fec04c50dc5b49878ffa3c3873df32e4' -> 'a14c220ad2cc4bf9bd7dad65f4803eb0' move completed: OK
I20240502 14:27:13.475119 21085 rebalancer_tool.cc:1136] tablet bad03a4ec8494938839d7d6eef8c684e: 'c902805060df4df9b5599d0e94b2dc06' -> '' move completed: OK
I20240502 14:27:13.623857 21085 rebalancer_tool.cc:1136] tablet 9580f3fdf7084cc0923b1b16bfedd450: '2db304b538e14857b7b1cc272d0db94b' -> 'a834edffbcb2469aae598b069aab0681' move completed: OK
I20240502 14:27:22.992497 21085 rebalancer_tool.cc:1136] tablet 414e03d597934da3a6a74d786aeab134: '007b2e79dc424a6f80141180f072f2d8' -> '7ab6d395b30d4afb9821b61c9454430b' move completed: OK
I20240502 14:27:23.288141 21085 rebalancer_tool.cc:1136] tablet df5f14206a1d46e4866f34fa234f945b: 'c902805060df4df9b5599d0e94b2dc06' -> 'a834edffbcb2469aae598b069aab0681' move completed: OK
I20240502 14:27:26.683969 21085 rebalancer_tool.cc:1136] tablet 96662a3d9e1b4f03b05464ddd06b93d6: '007b2e79dc424a6f80141180f072f2d8' -> 'a14c220ad2cc4bf9bd7dad65f4803eb0' move completed: OK
I20240502 14:27:27.377729 21085 rebalancer_tool.cc:1074] tablet 51ef0be93c914e258635e13f3cca79aa: '2db304b538e14857b7b1cc272d0db94b' -> '7ab6d395b30d4afb9821b61c9454430b' move scheduled
I20240502 14:27:27.428437 21085 rebalancer_tool.cc:1074] tablet 6a992d12aee7416096d1353533397386: 'c902805060df4df9b5599d0e94b2dc06' -> '' move scheduled
I20240502 14:27:27.523787 21085 rebalancer_tool.cc:1136] tablet 6a992d12aee7416096d1353533397386: 'c902805060df4df9b5599d0e94b2dc06' -> '' move completed: OK
I20240502 14:27:27.674137 21085 rebalancer_tool.cc:1136] tablet 51ef0be93c914e258635e13f3cca79aa: '2db304b538e14857b7b1cc272d0db94b' -> '7ab6d395b30d4afb9821b61c9454430b' move completed: OK
I20240502 14:27:29.719905 21085 rebalancer_tool.cc:1136] tablet cf2a6426ae6c438f942d39530da6e3e3: '007b2e79dc424a6f80141180f072f2d8' -> '' move completed: OK
I20240502 14:27:29.805424 21085 rebalancer_tool.cc:1136] tablet b13bcd6e693f4c7687a1aabdafbb83c6: '2db304b538e14857b7b1cc272d0db94b' -> '' move completed: OK
I20240502 14:27:29.988026 21085 rebalancer_tool.cc:1136] tablet 0d8667d91fd64809b635362c4a8fa33f: '007b2e79dc424a6f80141180f072f2d8' -> '7ab6d395b30d4afb9821b61c9454430b' move completed: OK
I20240502 14:27:30.256810 21085 rebalancer_tool.cc:1136] tablet e5fb3b049ba94af998791640bd0e31c5: 'fec04c50dc5b49878ffa3c3873df32e4' -> 'a14c220ad2cc4bf9bd7dad65f4803eb0' move completed: OK
I20240502 14:27:30.349572 21085 rebalancer_tool.cc:1136] tablet 8937c809610542d88fdde3daf002c890: 'fec04c50dc5b49878ffa3c3873df32e4' -> '7ab6d395b30d4afb9821b61c9454430b' move completed: OK
I20240502 14:27:30.509117 21085 rebalancer_tool.cc:1136] tablet 097eb705c9e24740a5be61bf3354f3d8: 'fec04c50dc5b49878ffa3c3873df32e4' -> '' move completed: OK
I20240502 14:27:30.665557 21085 rebalancer_tool.cc:1136] tablet 0ffb6e582ef04fb38dd273c13febac58: 'c902805060df4df9b5599d0e94b2dc06' -> 'ff6acba046404ef0a6f41ad5e09d852f' move completed: OK
I20240502 14:27:30.803100 21085 rebalancer_tool.cc:1136] tablet 246a8425400f4890a5264c4ae8a627c5: '2db304b538e14857b7b1cc272d0db94b' -> 'a14c220ad2cc4bf9bd7dad65f4803eb0' move completed: OK
I20240502 14:27:30.905179 21085 rebalancer_tool.cc:1136] tablet 522014370f0a4cc3bd31aa00d6a05c5e: '2db304b538e14857b7b1cc272d0db94b' -> 'ff6acba046404ef0a6f41ad5e09d852f' move completed: OK
I20240502 14:27:31.019645 21085 rebalancer_tool.cc:1136] tablet 8b8d81c275144f898dcc69a6d98d11e0: '2db304b538e14857b7b1cc272d0db94b' -> 'ff6acba046404ef0a6f41ad5e09d852f' move completed: OK
I20240502 14:27:31.441134 21085 rebalancer_tool.cc:1136] tablet 2c2afaffde96462eacd0d32bbdec23b1: 'c902805060df4df9b5599d0e94b2dc06' -> 'a834edffbcb2469aae598b069aab0681' move completed: OK
I20240502 14:27:31.482422 21085 rebalancer_tool.cc:1074] tablet 67827c0262724897a05bf80df579bbf3: '2db304b538e14857b7b1cc272d0db94b' -> '' move scheduled
I20240502 14:27:31.615746 21085 rebalancer_tool.cc:1074] tablet b315327141954b53a0f87c2d1dc14b9b: '007b2e79dc424a6f80141180f072f2d8' -> '' move scheduled
I20240502 14:27:31.717070 21085 rebalancer_tool.cc:1136] tablet b315327141954b53a0f87c2d1dc14b9b: '007b2e79dc424a6f80141180f072f2d8' -> '' move completed: OK
I20240502 14:27:31.870879 21085 rebalancer_tool.cc:1136] tablet 67827c0262724897a05bf80df579bbf3: '2db304b538e14857b7b1cc272d0db94b' -> '' move completed: OK
I20240502 14:27:33.052415 21085 rebalancer_tool.cc:1136] tablet 1ecc6539b85d40e49df7ffc5f93cb801: '007b2e79dc424a6f80141180f072f2d8' -> '' move completed: OK
Illegal state: tablet server a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): unacceptable health status UNAVAILABLE
I20240502 14:27:34.781431 29417 consensus_queue.cc:1035] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:27:34.785512 19097 raft_consensus.cc:1261] T df5f14206a1d46e4866f34fa234f945b P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:27:34.786234 29417 consensus_queue.cc:1035] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 14:27:34.794152 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:27:34.794304 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:27:34.810043 29417 raft_consensus.cc:2932] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
I20240502 14:27:34.815687 19093 raft_consensus.cc:2932] T df5f14206a1d46e4866f34fa234f945b P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
I20240502 14:27:34.823261 18556 catalog_manager.cc:5484] T df5f14206a1d46e4866f34fa234f945b P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: config changed from index 5 to 6, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) added. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
I20240502 14:27:34.842528 20626 raft_consensus.cc:2932] T df5f14206a1d46e4866f34fa234f945b P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
W20240502 14:27:34.860497 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:27:34.864056 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:27:34.894882 19097 consensus_queue.cc:237] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:27:34.897064 20626 raft_consensus.cc:1261] T 097eb705c9e24740a5be61bf3354f3d8 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:27:34.899192 29339 consensus_queue.cc:1035] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:27:34.908620 29737 raft_consensus.cc:2932] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:34.909642 20626 raft_consensus.cc:2932] T 097eb705c9e24740a5be61bf3354f3d8 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:34.916324 18554 catalog_manager.cc:5484] T 097eb705c9e24740a5be61bf3354f3d8 P a834edffbcb2469aae598b069aab0681 reported cstate change: config changed from index 7 to 8, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:27:34.925227 18681 tablet_service.cc:1514] Processing DeleteTablet for tablet 097eb705c9e24740a5be61bf3354f3d8 with delete_type TABLET_DATA_TOMBSTONED (TS 2db304b538e14857b7b1cc272d0db94b not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:54454
I20240502 14:27:34.931524 29741 tablet_replica.cc:330] T 097eb705c9e24740a5be61bf3354f3d8 P 2db304b538e14857b7b1cc272d0db94b: stopping tablet replica
I20240502 14:27:34.931934 29741 raft_consensus.cc:2229] T 097eb705c9e24740a5be61bf3354f3d8 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:27:34.932327 29741 raft_consensus.cc:2258] T 097eb705c9e24740a5be61bf3354f3d8 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:27:34.933451 29741 ts_tablet_manager.cc:1875] T 097eb705c9e24740a5be61bf3354f3d8 P 2db304b538e14857b7b1cc272d0db94b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:27:34.938129 19097 consensus_queue.cc:237] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
W20240502 14:27:34.941238 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 097eb705c9e24740a5be61bf3354f3d8. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:27:34.941912 20626 raft_consensus.cc:1261] T 097eb705c9e24740a5be61bf3354f3d8 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:27:34.944061 29741 ts_tablet_manager.cc:1888] T 097eb705c9e24740a5be61bf3354f3d8 P 2db304b538e14857b7b1cc272d0db94b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 14:27:34.944419 29739 consensus_queue.cc:1035] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:27:34.944288 29741 log.cc:1196] T 097eb705c9e24740a5be61bf3354f3d8 P 2db304b538e14857b7b1cc272d0db94b: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal/wals/097eb705c9e24740a5be61bf3354f3d8
I20240502 14:27:34.947324 18544 catalog_manager.cc:4830] TS 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749): tablet 097eb705c9e24740a5be61bf3354f3d8 (table rebalance_test_table_4 [id=4dd18a179f134d1babc05959227a25cc]) successfully deleted
I20240502 14:27:34.948473 29739 raft_consensus.cc:2932] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:34.949064 20626 raft_consensus.cc:2932] T 097eb705c9e24740a5be61bf3354f3d8 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:34.954336 18546 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 097eb705c9e24740a5be61bf3354f3d8 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:27:34.957549 18554 catalog_manager.cc:5484] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: config changed from index 8 to 9, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
W20240502 14:27:34.966296 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:27:35.073706 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:35.085546 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:35.145038 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:35.151654 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:35.186204 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:35.190850 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:27:35.210289 29800 ts_tablet_manager.cc:902] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06: Initiating tablet copy from peer 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
W20240502 14:27:35.222200 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:27:35.253684 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:35.254348 29800 tablet_copy_client.cc:308] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.65:36749
I20240502 14:27:35.257160 18721 tablet_copy_service.cc:140] P 2db304b538e14857b7b1cc272d0db94b: Received BeginTabletCopySession request for tablet df5f14206a1d46e4866f34fa234f945b from peer c902805060df4df9b5599d0e94b2dc06 ({username='slave'} at 127.5.107.67:55959)
I20240502 14:27:35.257375 18721 tablet_copy_service.cc:161] P 2db304b538e14857b7b1cc272d0db94b: Beginning new tablet copy session on tablet df5f14206a1d46e4866f34fa234f945b from peer c902805060df4df9b5599d0e94b2dc06 at {username='slave'} at 127.5.107.67:55959: session id = c902805060df4df9b5599d0e94b2dc06-df5f14206a1d46e4866f34fa234f945b
I20240502 14:27:35.261374 18721 tablet_copy_source_session.cc:205] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:27:35.276372 29800 ts_tablet_manager.cc:1875] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:27:35.303601 29800 ts_tablet_manager.cc:1888] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 14:27:35.303942 29800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df5f14206a1d46e4866f34fa234f945b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:27:35.312047 29800 tablet_copy_client.cc:783] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06: tablet copy: Starting download of 0 data blocks...
I20240502 14:27:35.312500 29800 tablet_copy_client.cc:647] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06: tablet copy: Starting download of 1 WAL segments...
I20240502 14:27:35.317890 29800 tablet_copy_client.cc:523] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:27:35.321888 29800 tablet_bootstrap.cc:492] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06: Bootstrap starting.
W20240502 14:27:35.347898 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:27:35.363104 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:27:35.414530 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240502 14:27:35.414608 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 35 similar messages]
W20240502 14:27:35.415792 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:27:35.432426 29800 tablet_bootstrap.cc:492] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:27:35.432996 29800 tablet_bootstrap.cc:492] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06: Bootstrap complete.
I20240502 14:27:35.433382 29800 ts_tablet_manager.cc:1367] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06: Time spent bootstrapping tablet: real 0.112s user 0.053s sys 0.002s
I20240502 14:27:35.434643 29800 raft_consensus.cc:348] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:27:35.434955 29800 raft_consensus.cc:729] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Initialized, Role: FOLLOWER
I20240502 14:27:35.435295 29800 consensus_queue.cc:260] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:27:35.437000 29800 ts_tablet_manager.cc:1398] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
W20240502 14:27:35.438345 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:27:35.438616 18721 tablet_copy_service.cc:342] P 2db304b538e14857b7b1cc272d0db94b: Request end of tablet copy session c902805060df4df9b5599d0e94b2dc06-df5f14206a1d46e4866f34fa234f945b received from {username='slave'} at 127.5.107.67:55959
I20240502 14:27:35.438743 18721 tablet_copy_service.cc:433] P 2db304b538e14857b7b1cc272d0db94b: ending tablet copy session c902805060df4df9b5599d0e94b2dc06-df5f14206a1d46e4866f34fa234f945b on tablet df5f14206a1d46e4866f34fa234f945b with peer c902805060df4df9b5599d0e94b2dc06
W20240502 14:27:35.440217 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 53 similar messages]
I20240502 14:27:35.450927 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:35.453361 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:35.469126 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:35.502172 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:35.502966 29850 ts_tablet_manager.cc:902] T 097eb705c9e24740a5be61bf3354f3d8 P ff6acba046404ef0a6f41ad5e09d852f: Initiating tablet copy from peer 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:27:35.530102 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:35.541002 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:35.554431 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:35.557536 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 43 similar messages]
I20240502 14:27:35.569296 29850 tablet_copy_client.cc:308] T 097eb705c9e24740a5be61bf3354f3d8 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.68:41463
I20240502 14:27:35.570181 19117 tablet_copy_service.cc:140] P 007b2e79dc424a6f80141180f072f2d8: Received BeginTabletCopySession request for tablet 097eb705c9e24740a5be61bf3354f3d8 from peer ff6acba046404ef0a6f41ad5e09d852f ({username='slave'} at 127.5.107.72:33829)
I20240502 14:27:35.570371 19117 tablet_copy_service.cc:161] P 007b2e79dc424a6f80141180f072f2d8: Beginning new tablet copy session on tablet 097eb705c9e24740a5be61bf3354f3d8 from peer ff6acba046404ef0a6f41ad5e09d852f at {username='slave'} at 127.5.107.72:33829: session id = ff6acba046404ef0a6f41ad5e09d852f-097eb705c9e24740a5be61bf3354f3d8
I20240502 14:27:35.574126 19117 tablet_copy_source_session.cc:205] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:27:35.576481 29850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 097eb705c9e24740a5be61bf3354f3d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:27:35.579347 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:35.583542 20562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 13 similar messages]
I20240502 14:27:35.593173 18963 raft_consensus.cc:1203] T df5f14206a1d46e4866f34fa234f945b P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 14:27:35.593317 29850 tablet_copy_client.cc:783] T 097eb705c9e24740a5be61bf3354f3d8 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Starting download of 0 data blocks...
I20240502 14:27:35.593711 29850 tablet_copy_client.cc:647] T 097eb705c9e24740a5be61bf3354f3d8 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Starting download of 1 WAL segments...
I20240502 14:27:35.596061 29850 tablet_copy_client.cc:523] T 097eb705c9e24740a5be61bf3354f3d8 P ff6acba046404ef0a6f41ad5e09d852f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 14:27:35.611658 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:27:35.618723 29850 tablet_bootstrap.cc:492] T 097eb705c9e24740a5be61bf3354f3d8 P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap starting.
I20240502 14:27:35.622454 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:35.623442 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:35.668005 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:35.695616 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:35.712225 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:27:35.735790 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:27:35.761373 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:27:35.786379 29850 tablet_bootstrap.cc:492] T 097eb705c9e24740a5be61bf3354f3d8 P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:27:35.786839 29850 tablet_bootstrap.cc:492] T 097eb705c9e24740a5be61bf3354f3d8 P ff6acba046404ef0a6f41ad5e09d852f: Bootstrap complete.
I20240502 14:27:35.787132 29850 ts_tablet_manager.cc:1367] T 097eb705c9e24740a5be61bf3354f3d8 P ff6acba046404ef0a6f41ad5e09d852f: Time spent bootstrapping tablet: real 0.169s user 0.047s sys 0.000s
I20240502 14:27:35.787861 29850 raft_consensus.cc:348] T 097eb705c9e24740a5be61bf3354f3d8 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:35.788142 29850 raft_consensus.cc:729] T 097eb705c9e24740a5be61bf3354f3d8 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6acba046404ef0a6f41ad5e09d852f, State: Initialized, Role: FOLLOWER
I20240502 14:27:35.788468 29850 consensus_queue.cc:260] T 097eb705c9e24740a5be61bf3354f3d8 P ff6acba046404ef0a6f41ad5e09d852f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:27:35.789137 29850 ts_tablet_manager.cc:1398] T 097eb705c9e24740a5be61bf3354f3d8 P ff6acba046404ef0a6f41ad5e09d852f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:27:35.790006 19117 tablet_copy_service.cc:342] P 007b2e79dc424a6f80141180f072f2d8: Request end of tablet copy session ff6acba046404ef0a6f41ad5e09d852f-097eb705c9e24740a5be61bf3354f3d8 received from {username='slave'} at 127.5.107.72:33829
I20240502 14:27:35.790132 19117 tablet_copy_service.cc:433] P 007b2e79dc424a6f80141180f072f2d8: ending tablet copy session ff6acba046404ef0a6f41ad5e09d852f-097eb705c9e24740a5be61bf3354f3d8 on tablet 097eb705c9e24740a5be61bf3354f3d8 with peer ff6acba046404ef0a6f41ad5e09d852f
W20240502 14:27:35.890967 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:27:35.919270 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:27:35.952809 21028 raft_consensus.cc:1203] T 097eb705c9e24740a5be61bf3354f3d8 P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 14:27:35.983330 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:35.994340 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:36.051928 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:36.052538 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:36.052742 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:36.107928 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:27:36.135888 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:36.194516 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:36.204066 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:27:36.223242 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:27:36.265401 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:36.278072 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:36.281237 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:36.301017 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:36.356177 21083 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskgPEJ6s/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 14:26:43.174908 21088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:26:43.175298 21088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:26:43.190378 21088 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:26:47.146317 21088 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 14:26:49.223418 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3670a8bef9694c029d2128006cc913d3: could not find any suitable replica to move from server fec04c50dc5b49878ffa3c3873df32e4 to server 7ab6d395b30d4afb9821b61c9454430b
W20240502 14:26:49.223839 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0e84c82d60e64dbd9149c5f9fb7ff3c8: could not find any suitable replica to move from server 2db304b538e14857b7b1cc272d0db94b to server ff6acba046404ef0a6f41ad5e09d852f
W20240502 14:26:49.223981 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a9acd50a500c4f84a41c8d9ba23ae431: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server a14c220ad2cc4bf9bd7dad65f4803eb0
W20240502 14:26:49.224123 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 27066b01c29c4a1380ee4753a1c036d3: could not find any suitable replica to move from server fec04c50dc5b49878ffa3c3873df32e4 to server ff6acba046404ef0a6f41ad5e09d852f
W20240502 14:26:49.224290 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4dd18a179f134d1babc05959227a25cc: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server a834edffbcb2469aae598b069aab0681
W20240502 14:26:49.224613 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 995bc008b9f94332bfa2c42a77e41bb2: could not find any suitable replica to move from server 2db304b538e14857b7b1cc272d0db94b to server ff6acba046404ef0a6f41ad5e09d852f
W20240502 14:26:49.224749 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff5cff7e35c746a5bc64cc06f18bf7d9: could not find any suitable replica to move from server 007b2e79dc424a6f80141180f072f2d8 to server a14c220ad2cc4bf9bd7dad65f4803eb0
W20240502 14:26:49.224905 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0e84c82d60e64dbd9149c5f9fb7ff3c8: could not find any suitable replica to move from server fec04c50dc5b49878ffa3c3873df32e4 to server 7ab6d395b30d4afb9821b61c9454430b
W20240502 14:26:49.225078 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 27066b01c29c4a1380ee4753a1c036d3: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server a834edffbcb2469aae598b069aab0681
W20240502 14:26:49.225248 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 210510c054044948932a30db816646aa: could not find any suitable replica to move from server 007b2e79dc424a6f80141180f072f2d8 to server ff6acba046404ef0a6f41ad5e09d852f
W20240502 14:26:49.225440 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5a8a0451cdf0479c94edf1d94e77fb83: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server 7ab6d395b30d4afb9821b61c9454430b
W20240502 14:26:49.225602 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b260896c8b8744c0a1b4f691850aec93: could not find any suitable replica to move from server fec04c50dc5b49878ffa3c3873df32e4 to server a14c220ad2cc4bf9bd7dad65f4803eb0
W20240502 14:26:49.225724 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3670a8bef9694c029d2128006cc913d3: could not find any suitable replica to move from server 007b2e79dc424a6f80141180f072f2d8 to server ff6acba046404ef0a6f41ad5e09d852f
W20240502 14:26:49.225884 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff5cff7e35c746a5bc64cc06f18bf7d9: could not find any suitable replica to move from server 2db304b538e14857b7b1cc272d0db94b to server 7ab6d395b30d4afb9821b61c9454430b
W20240502 14:26:49.226017 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 995bc008b9f94332bfa2c42a77e41bb2: could not find any suitable replica to move from server fec04c50dc5b49878ffa3c3873df32e4 to server a14c220ad2cc4bf9bd7dad65f4803eb0
I20240502 14:26:49.539701 21088 rebalancer_tool.cc:1074] tablet 8937c809610542d88fdde3daf002c890: 'c902805060df4df9b5599d0e94b2dc06' -> 'ff6acba046404ef0a6f41ad5e09d852f' move scheduled
I20240502 14:26:49.854574 21088 rebalancer_tool.cc:1084] tablet cf2a6426ae6c438f942d39530da6e3e3: '2db304b538e14857b7b1cc272d0db94b' -> 'a14c220ad2cc4bf9bd7dad65f4803eb0' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:26:49.854820 21088 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:26:52.903540 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 995bc008b9f94332bfa2c42a77e41bb2: could not find any suitable replica to move from server fec04c50dc5b49878ffa3c3873df32e4 to server 7ab6d395b30d4afb9821b61c9454430b
W20240502 14:26:52.903934 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b260896c8b8744c0a1b4f691850aec93: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server ff6acba046404ef0a6f41ad5e09d852f
W20240502 14:26:52.904193 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4dd18a179f134d1babc05959227a25cc: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server 7ab6d395b30d4afb9821b61c9454430b
W20240502 14:26:52.904317 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5a8a0451cdf0479c94edf1d94e77fb83: could not find any suitable replica to move from server fec04c50dc5b49878ffa3c3873df32e4 to server a834edffbcb2469aae598b069aab0681
W20240502 14:26:52.904426 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff5cff7e35c746a5bc64cc06f18bf7d9: could not find any suitable replica to move from server 007b2e79dc424a6f80141180f072f2d8 to server a834edffbcb2469aae598b069aab0681
W20240502 14:26:52.904600 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a9acd50a500c4f84a41c8d9ba23ae431: could not find any suitable replica to move from server 2db304b538e14857b7b1cc272d0db94b to server ff6acba046404ef0a6f41ad5e09d852f
W20240502 14:26:52.904749 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5a8a0451cdf0479c94edf1d94e77fb83: could not find any suitable replica to move from server 007b2e79dc424a6f80141180f072f2d8 to server a14c220ad2cc4bf9bd7dad65f4803eb0
W20240502 14:26:52.904920 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3670a8bef9694c029d2128006cc913d3: could not find any suitable replica to move from server fec04c50dc5b49878ffa3c3873df32e4 to server a14c220ad2cc4bf9bd7dad65f4803eb0
W20240502 14:26:52.905019 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 995bc008b9f94332bfa2c42a77e41bb2: could not find any suitable replica to move from server 2db304b538e14857b7b1cc272d0db94b to server a834edffbcb2469aae598b069aab0681
W20240502 14:26:52.905139 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0e84c82d60e64dbd9149c5f9fb7ff3c8: could not find any suitable replica to move from server 007b2e79dc424a6f80141180f072f2d8 to server ff6acba046404ef0a6f41ad5e09d852f
W20240502 14:26:52.905301 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b260896c8b8744c0a1b4f691850aec93: could not find any suitable replica to move from server fec04c50dc5b49878ffa3c3873df32e4 to server 7ab6d395b30d4afb9821b61c9454430b
W20240502 14:26:52.905437 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 210510c054044948932a30db816646aa: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server 7ab6d395b30d4afb9821b61c9454430b
W20240502 14:26:52.905561 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff5cff7e35c746a5bc64cc06f18bf7d9: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server a834edffbcb2469aae598b069aab0681
W20240502 14:26:52.905714 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4dd18a179f134d1babc05959227a25cc: could not find any suitable replica to move from server fec04c50dc5b49878ffa3c3873df32e4 to server ff6acba046404ef0a6f41ad5e09d852f
W20240502 14:26:52.905827 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 27066b01c29c4a1380ee4753a1c036d3: could not find any suitable replica to move from server 2db304b538e14857b7b1cc272d0db94b to server a14c220ad2cc4bf9bd7dad65f4803eb0
I20240502 14:26:52.953051 21088 rebalancer_tool.cc:1074] tablet 9580f3fdf7084cc0923b1b16bfedd450: '2db304b538e14857b7b1cc272d0db94b' -> '' move scheduled
I20240502 14:26:53.978816 21088 rebalancer_tool.cc:1074] tablet 097eb705c9e24740a5be61bf3354f3d8: 'fec04c50dc5b49878ffa3c3873df32e4' -> 'a834edffbcb2469aae598b069aab0681' move scheduled
I20240502 14:26:54.075804 21088 rebalancer_tool.cc:1084] tablet bad03a4ec8494938839d7d6eef8c684e: '007b2e79dc424a6f80141180f072f2d8' -> 'a14c220ad2cc4bf9bd7dad65f4803eb0' move ignored: Illegal state: tablet bad03a4ec8494938839d7d6eef8c684e: replica a14c220ad2cc4bf9bd7dad65f4803eb0 is already present
I20240502 14:26:54.075947 21088 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:26:55.759362 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a9acd50a500c4f84a41c8d9ba23ae431: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server 7ab6d395b30d4afb9821b61c9454430b
W20240502 14:26:55.759651 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4dd18a179f134d1babc05959227a25cc: could not find any suitable replica to move from server 2db304b538e14857b7b1cc272d0db94b to server ff6acba046404ef0a6f41ad5e09d852f
W20240502 14:26:55.759878 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff5cff7e35c746a5bc64cc06f18bf7d9: could not find any suitable replica to move from server 007b2e79dc424a6f80141180f072f2d8 to server 7ab6d395b30d4afb9821b61c9454430b
W20240502 14:26:55.759990 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a9acd50a500c4f84a41c8d9ba23ae431: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server a14c220ad2cc4bf9bd7dad65f4803eb0
W20240502 14:26:55.760269 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 27066b01c29c4a1380ee4753a1c036d3: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server a834edffbcb2469aae598b069aab0681
W20240502 14:26:55.760403 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3670a8bef9694c029d2128006cc913d3: could not find any suitable replica to move from server fec04c50dc5b49878ffa3c3873df32e4 to server 7ab6d395b30d4afb9821b61c9454430b
W20240502 14:26:55.760504 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 210510c054044948932a30db816646aa: could not find any suitable replica to move from server 2db304b538e14857b7b1cc272d0db94b to server a14c220ad2cc4bf9bd7dad65f4803eb0
W20240502 14:26:55.760593 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0e84c82d60e64dbd9149c5f9fb7ff3c8: could not find any suitable replica to move from server fec04c50dc5b49878ffa3c3873df32e4 to server a834edffbcb2469aae598b069aab0681
W20240502 14:26:55.760695 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ff5cff7e35c746a5bc64cc06f18bf7d9: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:26:56.379779 21088 rebalancer_tool.cc:1074] tablet 67827c0262724897a05bf80df579bbf3: '2db304b538e14857b7b1cc272d0db94b' -> 'a14c220ad2cc4bf9bd7dad65f4803eb0' move scheduled
I20240502 14:26:56.605070 21088 rebalancer_tool.cc:1074] tablet 522014370f0a4cc3bd31aa00d6a05c5e: '007b2e79dc424a6f80141180f072f2d8' -> '' move scheduled
I20240502 14:26:57.078754 21088 rebalancer_tool.cc:1074] tablet ecb8a9b04ba34f779e2227e77e6f764c: 'fec04c50dc5b49878ffa3c3873df32e4' -> '7ab6d395b30d4afb9821b61c9454430b' move scheduled
I20240502 14:26:57.511835 21088 rebalancer_tool.cc:1074] tablet 6a992d12aee7416096d1353533397386: 'c902805060df4df9b5599d0e94b2dc06' -> 'a834edffbcb2469aae598b069aab0681' move scheduled
I20240502 14:26:57.656519 21088 rebalancer_tool.cc:1084] tablet 1ecc6539b85d40e49df7ffc5f93cb801: '2db304b538e14857b7b1cc272d0db94b' -> 'ff6acba046404ef0a6f41ad5e09d852f' move ignored: Illegal state: tablet 1ecc6539b85d40e49df7ffc5f93cb801: replica ff6acba046404ef0a6f41ad5e09d852f is already present
I20240502 14:26:57.656697 21088 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:26:59.884507 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5a8a0451cdf0479c94edf1d94e77fb83: could not find any suitable replica to move from server 007b2e79dc424a6f80141180f072f2d8 to server ff6acba046404ef0a6f41ad5e09d852f
W20240502 14:26:59.884677 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 27066b01c29c4a1380ee4753a1c036d3: could not find any suitable replica to move from server c902805060df4df9b5599d0e94b2dc06 to server a14c220ad2cc4bf9bd7dad65f4803eb0
W20240502 14:26:59.884928 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b260896c8b8744c0a1b4f691850aec93: could not find any suitable replica to move from server fec04c50dc5b49878ffa3c3873df32e4 to server ff6acba046404ef0a6f41ad5e09d852f
W20240502 14:26:59.885067 21088 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a9acd50a500c4f84a41c8d9ba23ae431: could not find any suitable replica to move from server 2db304b538e14857b7b1cc272d0db94b to server a834edffbcb2469aae598b069aab0681
I20240502 14:26:59.981762 21088 rebalancer_tool.cc:1074] tablet 8a2bed29fcd64f7bbb04286404ef09de: '007b2e79dc424a6f80141180f072f2d8' -> '' move scheduled
I20240502 14:27:00.052888 21088 rebalancer_tool.cc:1074] tablet 21af917743444f2db932636a78f33512: 'fec04c50dc5b49878ffa3c3873df32e4' -> '' move scheduled
I20240502 14:27:00.211985 21088 rebalancer_tool.cc:1074] tablet 1ecc6539b85d40e49df7ffc5f93cb801: 'c902805060df4df9b5599d0e94b2dc06' -> '' move scheduled
I20240502 14:27:02.169567 21088 rebalancer_tool.cc:1123] tablet 1ecc6539b85d40e49df7ffc5f93cb801: 'c902805060df4df9b5599d0e94b2dc06' -> '' move is abandoned: Illegal state: Replica 007b2e79dc424a6f80141180f072f2d8 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:27:14.692627 21088 rebalancer_tool.cc:1136] tablet 67827c0262724897a05bf80df579bbf3: '2db304b538e14857b7b1cc272d0db94b' -> 'a14c220ad2cc4bf9bd7dad65f4803eb0' move completed: OK
I20240502 14:27:21.072679 21088 rebalancer_tool.cc:1136] tablet 21af917743444f2db932636a78f33512: 'fec04c50dc5b49878ffa3c3873df32e4' -> '' move completed: OK
I20240502 14:27:21.219434 21088 rebalancer_tool.cc:1136] tablet 8a2bed29fcd64f7bbb04286404ef09de: '007b2e79dc424a6f80141180f072f2d8' -> '' move completed: OK
I20240502 14:27:21.287433 21088 rebalancer_tool.cc:1136] tablet 8937c809610542d88fdde3daf002c890: 'c902805060df4df9b5599d0e94b2dc06' -> 'ff6acba046404ef0a6f41ad5e09d852f' move completed: OK
I20240502 14:27:21.447851 21088 rebalancer_tool.cc:1136] tablet 9580f3fdf7084cc0923b1b16bfedd450: '2db304b538e14857b7b1cc272d0db94b' -> '' move completed: OK
I20240502 14:27:21.591583 21088 rebalancer_tool.cc:1136] tablet 097eb705c9e24740a5be61bf3354f3d8: 'fec04c50dc5b49878ffa3c3873df32e4' -> 'a834edffbcb2469aae598b069aab0681' move completed: OK
I20240502 14:27:21.665776 21088 rebalancer_tool.cc:1136] tablet 6a992d12aee7416096d1353533397386: 'c902805060df4df9b5599d0e94b2dc06' -> 'a834edffbcb2469aae598b069aab0681' move completed: OK
I20240502 14:27:21.825158 21088 rebalancer_tool.cc:1136] tablet ecb8a9b04ba34f779e2227e77e6f764c: 'fec04c50dc5b49878ffa3c3873df32e4' -> '7ab6d395b30d4afb9821b61c9454430b' move completed: OK
I20240502 14:27:22.034984 21088 rebalancer_tool.cc:1136] tablet 522014370f0a4cc3bd31aa00d6a05c5e: '007b2e79dc424a6f80141180f072f2d8' -> '' move completed: OK
I20240502 14:27:22.094980 21088 rebalancer_tool.cc:1074] tablet cf2a6426ae6c438f942d39530da6e3e3: '2db304b538e14857b7b1cc272d0db94b' -> '' move scheduled
I20240502 14:27:22.173447 21088 rebalancer_tool.cc:1074] tablet 0ffb6e582ef04fb38dd273c13febac58: 'fec04c50dc5b49878ffa3c3873df32e4' -> '' move scheduled
I20240502 14:27:22.281322 21088 rebalancer_tool.cc:1074] tablet 414e03d597934da3a6a74d786aeab134: '007b2e79dc424a6f80141180f072f2d8' -> '' move scheduled
I20240502 14:27:22.377992 21088 rebalancer_tool.cc:1074] tablet bad03a4ec8494938839d7d6eef8c684e: 'c902805060df4df9b5599d0e94b2dc06' -> '' move scheduled
I20240502 14:27:22.459393 21088 rebalancer_tool.cc:1136] tablet bad03a4ec8494938839d7d6eef8c684e: 'c902805060df4df9b5599d0e94b2dc06' -> '' move completed: OK
I20240502 14:27:22.734953 21088 rebalancer_tool.cc:1136] tablet 414e03d597934da3a6a74d786aeab134: '007b2e79dc424a6f80141180f072f2d8' -> '' move completed: OK
I20240502 14:27:22.822908 21088 rebalancer_tool.cc:1136] tablet 0ffb6e582ef04fb38dd273c13febac58: 'fec04c50dc5b49878ffa3c3873df32e4' -> '' move completed: OK
I20240502 14:27:22.934609 21088 rebalancer_tool.cc:1136] tablet cf2a6426ae6c438f942d39530da6e3e3: '2db304b538e14857b7b1cc272d0db94b' -> '' move completed: OK
I20240502 14:27:23.067658 21088 rebalancer_tool.cc:1074] tablet b15d11209e45473e863766e5cd6e822f: '007b2e79dc424a6f80141180f072f2d8' -> '' move scheduled
I20240502 14:27:23.260785 21088 rebalancer_tool.cc:1074] tablet e5fb3b049ba94af998791640bd0e31c5: 'fec04c50dc5b49878ffa3c3873df32e4' -> '' move scheduled
I20240502 14:27:23.438555 21088 rebalancer_tool.cc:1074] tablet 0d8667d91fd64809b635362c4a8fa33f: '2db304b538e14857b7b1cc272d0db94b' -> '' move scheduled
I20240502 14:27:23.582152 21088 rebalancer_tool.cc:1074] tablet 5be0d6f4f1874c79b906c4bdc34b0ce9: 'c902805060df4df9b5599d0e94b2dc06' -> '' move scheduled
I20240502 14:27:23.681354 21088 rebalancer_tool.cc:1136] tablet 5be0d6f4f1874c79b906c4bdc34b0ce9: 'c902805060df4df9b5599d0e94b2dc06' -> '' move completed: OK
I20240502 14:27:23.765928 21088 rebalancer_tool.cc:1136] tablet 0d8667d91fd64809b635362c4a8fa33f: '2db304b538e14857b7b1cc272d0db94b' -> '' move completed: OK
I20240502 14:27:23.908730 21088 rebalancer_tool.cc:1136] tablet e5fb3b049ba94af998791640bd0e31c5: 'fec04c50dc5b49878ffa3c3873df32e4' -> '' move completed: OK
I20240502 14:27:27.474462 21088 rebalancer_tool.cc:1074] tablet 2c2afaffde96462eacd0d32bbdec23b1: '2db304b538e14857b7b1cc272d0db94b' -> '' move scheduled
I20240502 14:27:27.583765 21088 rebalancer_tool.cc:1074] tablet 332e4b23751b422ab7bd17b80c93701d: '007b2e79dc424a6f80141180f072f2d8' -> '' move scheduled
I20240502 14:27:27.658985 21088 rebalancer_tool.cc:1074] tablet 6acde0e5509340fbb52bca18645a77b7: 'c902805060df4df9b5599d0e94b2dc06' -> '' move scheduled
I20240502 14:27:27.677940 21088 rebalancer_tool.cc:1136] tablet 6acde0e5509340fbb52bca18645a77b7: 'c902805060df4df9b5599d0e94b2dc06' -> '' move completed: OK
I20240502 14:27:27.733875 21088 rebalancer_tool.cc:1136] tablet 332e4b23751b422ab7bd17b80c93701d: '007b2e79dc424a6f80141180f072f2d8' -> '' move completed: OK
I20240502 14:27:27.873915 21088 rebalancer_tool.cc:1136] tablet 2c2afaffde96462eacd0d32bbdec23b1: '2db304b538e14857b7b1cc272d0db94b' -> '' move completed: OK
I20240502 14:27:28.063977 21088 rebalancer_tool.cc:1136] tablet b15d11209e45473e863766e5cd6e822f: '007b2e79dc424a6f80141180f072f2d8' -> '' move completed: OK
I20240502 14:27:28.203576 21088 rebalancer_tool.cc:1074] tablet b13bcd6e693f4c7687a1aabdafbb83c6: '2db304b538e14857b7b1cc272d0db94b' -> '' move scheduled
I20240502 14:27:28.278483 21088 rebalancer_tool.cc:1136] tablet b13bcd6e693f4c7687a1aabdafbb83c6: '2db304b538e14857b7b1cc272d0db94b' -> '' move completed: OK
I20240502 14:27:30.237771 21088 rebalancer_tool.cc:1074] tablet d28db16c9f7e405b8866f0e6c548e246: '7ab6d395b30d4afb9821b61c9454430b' -> '007b2e79dc424a6f80141180f072f2d8' move scheduled
I20240502 14:27:32.278357 21088 rebalancer_tool.cc:1074] tablet 6a992d12aee7416096d1353533397386: 'a14c220ad2cc4bf9bd7dad65f4803eb0' -> '2db304b538e14857b7b1cc272d0db94b' move scheduled
I20240502 14:27:32.815488 21088 rebalancer_tool.cc:1074] tablet db56136f2dbb429098d4d78b15a3a439: 'a834edffbcb2469aae598b069aab0681' -> 'fec04c50dc5b49878ffa3c3873df32e4' move scheduled
I20240502 14:27:34.229573 21088 rebalancer_tool.cc:1074] tablet 0ffb6e582ef04fb38dd273c13febac58: 'ff6acba046404ef0a6f41ad5e09d852f' -> 'c902805060df4df9b5599d0e94b2dc06' move scheduled
I20240502 14:27:34.305694 21088 rebalancer_tool.cc:1074] tablet 96662a3d9e1b4f03b05464ddd06b93d6: 'a14c220ad2cc4bf9bd7dad65f4803eb0' -> '' move scheduled
I20240502 14:27:34.825316 21088 rebalancer_tool.cc:1074] tablet df5f14206a1d46e4866f34fa234f945b: '007b2e79dc424a6f80141180f072f2d8' -> 'c902805060df4df9b5599d0e94b2dc06' move scheduled
I20240502 14:27:34.988422 21088 rebalancer_tool.cc:1084] tablet 097eb705c9e24740a5be61bf3354f3d8: 'a834edffbcb2469aae598b069aab0681' -> 'ff6acba046404ef0a6f41ad5e09d852f' move ignored: Illegal state: Request specified cas_config_opid_index of 7 but the committed config has opid_index of 9
I20240502 14:27:34.988866 21088 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): unacceptable health status UNAVAILABLE
W20240502 14:27:36.508769 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:27:36.571210 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:36.592479 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:36.632675 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:36.640136 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:36.640158 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:27:36.662349 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:27:36.668543 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:36.682832 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:27:36.689486 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:27:36.707022 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:27:36.709797 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:36.740445 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:27:36.776494 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:36.788444 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:27:36.796273 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:36.831192 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:36.869832 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:36.876896 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:36.888713 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:36.962041 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:27:37.007407 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:27:37.078076 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:37.078976 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:37.080118 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:37.084111 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:27:37.087339 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:37.091178 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:27:37.099694 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:27:37.100841 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:37.110878 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:37.153033 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:27:37.159287 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:27:37.179244 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:27:37.200702 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:27:37.252503 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:27:37.292613 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:27:37.319361 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:27:37.332410 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:37.339589 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:37.355682 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:37.358716 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:27:37.373690 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:37.408020 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:37.426198 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:37.438150 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:27:37.451081 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:27:37.460531 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:27:37.508839 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:27:37.521816 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:27:37.525844 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:37.542438 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:37.544692 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:37.555925 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:37.593696 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:37.606608 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:27:37.747375 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:27:37.846333 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:27:37.852113 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:27:37.860749 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:37.899428 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:37.920015 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:37.957476 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:37.960139 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:38.004141 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:27:38.063293 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:38.076342 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:27:38.095467 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:27:38.097246 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:38.103127 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:38.105315 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:38.105688 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:38.162043 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:27:38.186328 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:27:38.219223 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:27:38.320768 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:27:38.353765 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:27:38.384011 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:38.389412 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:38.428856 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:38.442250 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:27:38.458170 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:38.460491 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:38.471262 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:38.486348 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:27:38.672351 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:27:38.709756 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:38.725828 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:38.761133 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:38.779724 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:38.810343 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:38.813804 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:38.854758 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:38.884142 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:38.905643 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:38.914863 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:27:38.941736 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:38.944682 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:38.979357 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:39.083779 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:27:39.119117 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:27:39.139628 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:27:39.145988 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:39.147315 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:39.153111 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:39.154620 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:39.155874 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:39.175138 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:39.234341 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:27:39.276638 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:27:39.311342 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:27:39.317312 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:27:39.326213 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:27:39.362259 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:27:39.397840 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:27:39.418954 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:27:39.438230 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:27:39.458329 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:39.458817 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:39.463644 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:27:39.479756 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:39.480314 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:39.484273 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:39.565841 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:27:39.570003 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:27:39.572185 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:27:39.595253 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:39.625267 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:39.625324 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:39.628214 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:27:39.654999 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:39.657418 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:39.657773 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:39.694785 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:27:39.730279 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:27:39.778949 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:27:39.826279 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:27:39.827359 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:39.830796 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:27:39.831844 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:27:39.854943 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:39.856420 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:39.874264 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:39.898280 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:39.909794 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:27:39.923202 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:27:39.927147 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:27:40.003440 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:27:40.106902 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:40.109206 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:40.111256 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:40.112910 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:40.116391 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:40.119904 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:40.151412 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:27:40.344612 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:27:40.363159 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:27:40.374110 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:27:40.399273 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:40.411679 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:40.418351 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:40.424703 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 14:27:40.425644 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:40.425628 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:27:40.433559 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:40.465008 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 14:27:40.469281 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240502 14:27:40.529991 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:27:40.531626 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:27:40.586427 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:40.622972 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:40.623530 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:40.637013 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:40.639420 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:40.661722 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 14:27:40.719461 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:27:40.729010 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:27:40.743400 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:27:40.777695 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:27:40.863873 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:27:40.874426 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:27:40.881382 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:40.904132 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:40.923794 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:27:40.936915 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:27:40.948707 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:40.964766 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:41.011521 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:41.041702 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:41.198169 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:41.214278 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:41.237877 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:41.251624 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:41.255126 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:41.258066 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:27:41.269431 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:41.485839 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:41.511418 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:41.512938 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:41.519555 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:41.530794 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:41.541599 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:41.584234 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:27:41.613162 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:27:41.620976 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:27:41.659826 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:27:41.662601 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:27:41.665154 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:27:41.754746 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:27:41.794164 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:41.804433 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:41.817322 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:27:41.823129 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:41.840188 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:41.851886 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:27:41.852041 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:27:41.853660 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:41.881083 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:41.885982 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:41.901573 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:41.928949 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:27:41.963115 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:41.966250 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:27:41.970348 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:41.975171 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:27:41.979331 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:27:41.981885 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:27:41.987311 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:42.007458 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:42.025975 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:27:42.083153 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:27:42.157981 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:27:42.161151 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:27:42.204183 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:42.207417 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:42.209367 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:42.225404 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:27:42.238380 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:42.258244 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:42.262260 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:42.273947 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:27:42.306881 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:27:42.444186 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:42.452780 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:42.469169 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:27:42.505540 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:42.508976 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:42.523125 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:42.586146 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:27:42.606050 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:27:42.616508 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:42.702359 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:27:42.736686 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:42.747987 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:42.751868 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:42.762372 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:42.766425 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:42.782490 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:42.816380 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:27:42.860765 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:27:42.957113 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:27:42.969336 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:43.003899 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:43.010675 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:43.015242 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:43.027966 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:43.040267 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:43.260332 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:43.265569 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:43.276461 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:43.281164 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:27:43.285342 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:43.290215 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:43.301349 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:43.330220 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:27:43.336522 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:27:43.338603 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:27:43.390400 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:27:43.456698 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:27:43.460276 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:27:43.483695 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:27:43.569947 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:43.582883 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:43.583331 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:43.612843 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:43.624586 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:43.628181 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:43.641273 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:43.670876 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:43.710952 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:27:43.713090 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:43.719368 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:43.731223 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:43.738359 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:43.846781 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:27:43.859459 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:27:44.005070 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:27:44.011031 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:27:44.037020 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:27:44.042588 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:44.052744 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:44.053782 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:44.095350 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:44.100404 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:44.100947 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:44.143625 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:44.151424 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:27:44.184604 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:44.188131 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:27:44.208340 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:27:44.237931 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:27:44.248617 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:44.265991 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:27:44.287384 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:44.293778 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:27:44.308104 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:44.308425 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:44.344931 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:27:44.400997 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:27:44.403695 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:27:44.462350 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:27:44.509498 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:27:44.649760 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:27:44.693677 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:27:44.704931 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:27:44.723637 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:27:44.739686 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:27:44.771248 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:44.773589 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:44.817474 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:44.839319 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:44.860013 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:44.875080 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:44.883073 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:27:44.896245 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:27:44.946375 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:44.957615 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:44.979393 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:44.996912 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:45.004400 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:45.022082 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:45.042541 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:27:45.216373 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:45.234376 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:27:45.238991 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:27:45.240660 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:27:45.247303 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:45.261065 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:45.272403 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:45.274844 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:45.341333 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:27:45.366390 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:27:45.393056 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:45.421408 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:45.443161 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 14:27:45.444219 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:27:45.449641 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:45.451992 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:45.469323 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:45.474237 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 89 similar messages]
I20240502 14:27:45.481814 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:45.516911 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240502 14:27:45.575418 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:27:45.603278 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:27:45.681910 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:45.702852 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:45.707123 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 14:27:45.709252 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:27:45.719523 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:45.730229 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:45.741869 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:45.749883 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:45.865155 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:27:45.866869 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:27:45.916925 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:27:45.958076 20562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:27:45.975065 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:27:45.978313 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:27:46.055955 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:46.061259 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:46.062878 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:46.080694 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:46.083081 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:46.089965 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:46.126232 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:27:46.166980 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:27:46.197842 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:27:46.265725 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:27:46.323823 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:27:46.412415 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:46.422502 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:46.445745 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:46.452133 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:46.472003 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:46.472856 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:27:46.480278 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:27:46.484834 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:46.496613 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:46.525089 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:46.526055 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:46.542429 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:27:46.564106 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:46.572685 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:46.576025 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:46.588502 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:27:46.631584 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:27:46.690035 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:27:46.790380 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:27:46.813004 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:27:46.813581 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:46.817556 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:27:46.823763 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:46.824790 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:46.827076 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:46.828652 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:46.861027 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:27:46.865537 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:27:46.954186 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:27:46.955006 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:27:46.962653 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:27:46.972362 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:27:46.988924 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:27:47.039039 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:47.060529 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:47.060611 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:47.074770 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:47.096571 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:47.108306 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:47.125674 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:27:47.224684 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:47.230553 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:47.258059 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:47.281589 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:27:47.287488 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:47.290231 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:47.305519 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:47.312767 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:27:47.374662 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:27:47.386473 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:27:47.407917 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:27:47.443753 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:47.452572 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:47.459695 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:47.463351 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:47.505016 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:47.516297 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:47.570922 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:27:47.720029 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:47.723752 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:47.725775 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:47.737752 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:47.737897 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:47.756065 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:47.787568 18768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.72:35871, user_credentials={real_user=slave}} blocked reactor thread for 45498.2us
W20240502 14:27:47.798312 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:27:47.823478 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:27:47.868551 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:27:47.941852 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:27:48.026464 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:48.031345 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:27:48.032829 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:27:48.056818 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:48.060477 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:48.069584 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:48.086467 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:48.088055 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:48.090341 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:48.096253 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:48.117503 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:48.158109 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:48.221459 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:48.244973 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:27:48.337744 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:27:48.362085 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:27:48.404073 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:27:48.455415 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:27:48.455483 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:48.466495 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:27:48.480508 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:27:48.480916 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:27:48.496189 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:27:48.522967 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:48.553057 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:48.565999 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:48.575994 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:48.619478 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:27:48.621268 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:48.621379 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:48.638411 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:48.639578 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:48.647017 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:48.710474 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:27:48.732277 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:27:48.827561 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:27:48.954342 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:27:49.069065 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:27:49.114888 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:27:49.154453 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:27:49.157745 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:49.163792 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:49.166816 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:49.172950 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:49.181473 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:49.216939 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:49.227317 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:27:49.227836 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:27:49.233074 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:49.262585 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:27:49.297696 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:27:49.298643 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:49.304464 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:49.335181 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:27:49.337007 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:27:49.337736 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:49.338207 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:49.339599 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:27:49.361869 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:27:49.375089 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:27:49.517369 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:27:49.557140 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:27:49.558478 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:27:49.559302 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:49.560181 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:49.596558 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:49.618549 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:49.638240 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:49.662904 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:49.671137 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:27:49.686007 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:27:49.702266 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:49.733274 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:27:49.795089 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:27:49.795219 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:49.796535 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:49.845131 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:49.862232 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:27:49.863453 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:49.880267 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:27:49.902418 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:27:50.047494 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:50.060405 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:50.065765 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:50.067827 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:50.072816 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:50.124321 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:50.139462 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:27:50.285298 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:50.294576 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:50.295804 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:50.310097 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:50.313489 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:50.325798 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:50.351418 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:27:50.353963 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:27:50.433813 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:27:50.447006 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:27:50.455469 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 14:27:50.472872 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:27:50.494271 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240502 14:27:50.495306 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:27:50.566179 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240502 14:27:50.608486 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:50.637843 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:50.640861 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:50.643494 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:50.644973 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:50.656714 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:50.665437 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:50.697945 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:50.716045 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 14:27:50.725942 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:50.733063 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:27:50.766484 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:27:50.774294 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:50.777707 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:50.785753 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:27:50.789121 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:50.832895 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:27:50.985823 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:51.004969 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:51.006547 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:51.015836 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:51.019439 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:51.038379 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:51.084635 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:27:51.088543 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:27:51.164847 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:27:51.188498 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:27:51.263386 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:27:51.277623 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:27:51.278591 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:51.310495 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:51.312001 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:51.316543 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:51.337508 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:51.361465 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:51.366132 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:27:51.378821 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:51.388839 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:51.394237 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:27:51.399665 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:51.416677 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:51.417873 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:51.472002 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:51.516309 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:27:51.535822 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:27:51.600435 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:27:51.616073 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:27:51.659006 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:51.687942 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:51.694475 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:51.696045 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:51.703459 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:51.718858 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:51.722186 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:27:51.773854 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:27:51.868433 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:27:51.886421 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:27:51.889048 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:51.908741 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:51.911680 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:51.926453 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:51.941483 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:51.954540 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:51.954735 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:27:51.967459 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:27:51.992502 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:27:51.993784 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:27:52.022280 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:27:52.050062 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:27:52.094799 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:27:52.114243 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:27:52.154728 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:27:52.161593 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:27:52.185227 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:27:52.212031 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:27:52.218183 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:52.241086 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:52.244803 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:52.256433 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:52.278604 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:52.295325 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:52.314635 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:52.327885 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:52.397279 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:52.402411 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:52.464525 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:52.479796 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:52.509076 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:27:52.642719 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:52.676118 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:52.683923 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:27:52.717456 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:52.747900 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:52.786290 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:52.822955 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:52.854207 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:27:52.890267 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:27:52.937657 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:52.940449 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:52.943620 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:52.947113 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:52.961616 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:52.974016 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:53.020123 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:27:53.065608 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:27:53.065999 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:27:53.213140 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:27:53.231693 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:27:53.260376 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:27:53.261145 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:53.264528 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:53.270035 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:53.277678 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:53.291309 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:53.296945 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:53.562223 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:27:53.573475 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:27:53.573988 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:27:53.577881 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:53.600530 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:27:53.605417 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:27:53.607261 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:53.622918 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:53.645731 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:53.658562 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:27:53.662082 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:27:53.668159 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:53.675650 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:53.693450 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:53.728622 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:53.729019 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:53.729153 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:53.761147 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:53.781095 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:27:53.822494 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:27:53.957435 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:53.962284 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:53.972469 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:53.983911 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:53.985054 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:54.002285 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:54.195245 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:27:54.209733 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:54.233901 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:27:54.238308 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:27:54.248123 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:27:54.253631 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:27:54.258549 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:54.279135 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:54.284536 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:27:54.292093 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:27:54.305786 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:54.309412 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:54.331797 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:54.337136 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:27:54.349921 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:27:54.355273 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:27:54.379514 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:54.380833 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:27:54.387522 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:27:54.389006 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:27:54.413916 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:54.434782 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:54.447100 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:54.459568 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:54.583294 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:27:54.594172 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:27:54.603375 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:27:54.672770 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:54.677320 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:54.686779 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:27:54.706125 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:54.716472 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:54.717576 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:54.722729 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:27:54.753350 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:54.792485 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:27:54.855655 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:27:54.901466 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:27:54.937690 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:54.943806 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:54.952577 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:54.964114 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:54.971907 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:54.975854 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:55.208880 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:55.225474 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:55.240408 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:27:55.267359 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:27:55.272122 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:27:55.302538 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:55.307155 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:55.311949 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:55.323846 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:55.351897 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:55.360424 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:55.392944 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:55.426241 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:55.430218 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:27:55.432703 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:27:55.467398 18899 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.72:35871, user_credentials={real_user=slave}} blocked reactor thread for 153306us
W20240502 14:27:55.491147 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:27:55.493314 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:55.510309 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:27:55.522954 19032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240502 14:27:55.548357 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:27:55.600251 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 14:27:55.647481 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 14:27:55.719071 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:27:55.741669 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:55.745043 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 14:27:55.757390 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:55.767477 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:55.769780 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:55.774607 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:55.775059 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:55.790007 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:27:55.900243 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:55.926273 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:55.937968 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:55.945298 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:55.947543 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:55.953729 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:55.971592 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:27:55.985088 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:27:56.004820 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:27:56.114648 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:27:56.227221 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:27:56.278048 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:27:56.279049 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:56.283041 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:56.284158 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:56.287974 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:56.305495 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:56.317445 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:56.378713 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:27:56.425726 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:27:56.431762 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:27:56.443869 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:27:56.497658 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:27:56.500830 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:27:56.509276 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:27:56.520010 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:56.537564 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:56.551599 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:27:56.563795 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:56.579340 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:56.591073 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:56.603627 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:27:56.654023 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:27:56.697654 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:27:56.746659 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:27:56.767596 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:27:56.775872 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:56.782435 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:56.793095 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:56.800652 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:56.831950 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:27:56.861124 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:56.868985 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:27:56.927774 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:27:56.979215 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:27:56.983044 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:56.985559 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:56.989243 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:56.991039 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:56.993654 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:27:57.021361 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:27:57.023075 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:27:57.024876 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:57.040975 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:27:57.048023 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:27:57.081665 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:27:57.136445 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:27:57.191658 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:27:57.233407 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:57.248577 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:57.299010 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:57.306232 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:57.306260 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:57.332268 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:27:57.365443 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:27:57.372103 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:57.383414 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:57.414497 20964 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.5.107.71:39195, user_credentials={real_user=slave}} blocked reactor thread for 59262.7us
I20240502 14:27:57.415110 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:57.417676 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:27:57.421818 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:57.423703 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:57.463708 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:57.580456 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:27:57.655692 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:57.669291 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:57.686843 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:57.687757 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:57.698434 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:57.700423 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:27:57.714365 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:57.772606 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:27:57.852541 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:27:57.910701 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:57.912727 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:57.922078 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:57.924310 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:27:57.925375 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:27:57.927958 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:57.930038 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:57.932978 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:57.981909 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:27:58.132570 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:27:58.202401 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:58.227331 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:27:58.248677 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:27:58.257225 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:58.260939 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:58.261377 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:58.294317 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:58.305963 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:58.311169 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:58.339147 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:27:58.371408 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:58.380494 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:58.392150 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:58.398612 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:58.546535 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:27:58.593474 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:27:58.678103 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:58.702047 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:58.711041 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:58.713750 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:27:58.720479 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:58.734815 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:58.750998 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:58.753463 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:58.763892 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:27:58.779484 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:58.793701 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:58.794348 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:27:58.811092 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:27:58.817488 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:27:58.831763 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:27:58.841122 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:58.884352 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:58.884759 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:58.888147 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:27:58.902629 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:27:59.051635 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:27:59.091938 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:27:59.140061 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:27:59.255815 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:27:59.317230 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:27:59.353372 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:27:59.360260 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:27:59.365698 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:27:59.377513 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:59.380324 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:27:59.398424 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:59.436023 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:27:59.459815 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:59.461885 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:59.464461 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:59.470664 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:27:59.526916 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:27:59.529995 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:27:59.629556 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:27:59.638095 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:59.638417 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:27:59.668015 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:27:59.684785 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:27:59.689733 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:59.690122 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:59.766556 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:27:59.781549 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:27:59.782652 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:27:59.866714 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:27:59.902253 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:27:59.931465 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:27:59.933499 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:27:59.935322 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:27:59.942147 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:27:59.947131 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:27:59.955229 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:28:00.002724 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:00.100888 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:00.118059 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:00.132565 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:28:00.156713 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:00.170773 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:00.180253 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:00.184782 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:00.189661 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:00.346400 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:00.350190 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:00.351794 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:00.361380 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:00.362710 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:28:00.390377 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:00.411635 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:00.412396 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:00.429503 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:28:00.448541 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:28:00.628365 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:00.631882 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:00.632093 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:00.636404 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:00.636998 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:00.649664 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:00.687577 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 14:28:00.719520 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 14:28:00.737846 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 14:28:00.753243 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 14:28:00.776126 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:00.801596 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:28:00.802561 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:00.843073 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:28:00.920563 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:00.926427 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:28:00.927034 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:00.928968 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:00.932848 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:28:00.951210 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:00.989521 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:00.992939 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:00.993918 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:01.013664 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:01.025646 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:01.053542 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:28:01.063185 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:01.068113 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:01.102856 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:01.246413 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:28:01.252671 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:01.262440 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:28:01.266531 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:01.273090 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:28:01.279932 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:01.283728 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:01.289975 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:01.293432 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:01.293852 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:01.332610 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:28:01.411933 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:01.501019 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:28:01.507210 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:28:01.543877 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:01.569790 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:01.598179 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:01.609388 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:01.624174 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:28:01.624441 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:28:01.634471 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:01.642297 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:28:01.656814 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:01.663704 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:01.679394 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:01.684226 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:01.702304 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:28:01.713055 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:28:01.720995 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:01.733688 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:01.765481 20562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 14:28:01.807301 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:01.808643 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:01.929271 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:28:01.973107 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:01.999420 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:02.000813 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:02.014529 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:02.022955 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:02.024509 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:02.033524 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:02.071187 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:28:02.081836 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:28:02.158684 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:28:02.186838 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:28:02.197134 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:02.220259 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:28:02.231935 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:02.237789 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:02.257262 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:02.266767 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:02.267555 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:02.282060 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:28:02.290975 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:02.408987 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:28:02.461269 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:28:02.477411 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:02.525676 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:02.559129 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:02.583721 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:02.632205 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:02.640345 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:02.666091 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:02.669301 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:02.704104 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:02.705904 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:02.723640 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:02.749081 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:28:02.759559 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:02.768791 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:02.786803 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:28:02.826226 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:02.880810 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:28:02.975181 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:28:03.089670 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:03.113930 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:03.115293 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:03.119714 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:03.137637 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:03.166586 20964 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.5.107.71:39195, user_credentials={real_user=slave}} blocked reactor thread for 93245.3us
W20240502 14:28:03.167804 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:03.178197 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:03.215200 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:28:03.221742 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:28:03.223264 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:03.255048 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:03.306768 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:03.309942 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:03.313645 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:03.323385 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:03.332731 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:03.352285 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:28:03.532217 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:03.550621 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:28:03.612478 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:03.628086 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:03.636862 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:03.641253 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:03.644431 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:03.654101 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:28:03.659689 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:03.708671 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:28:03.859377 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:28:03.863420 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:03.885380 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:28:03.912863 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:28:03.931258 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:03.943534 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:03.991781 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:03.998590 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:03.999644 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:04.003927 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:04.036605 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:04.058954 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:04.084508 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:04.087257 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:04.099642 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:04.110396 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:04.123116 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:28:04.144577 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:04.197604 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:28:04.217038 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:28:04.232367 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:28:04.238144 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:04.271888 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:28:04.328612 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:04.338968 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:04.351258 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:28:04.357506 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:04.358294 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:04.359807 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:04.360160 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:04.404018 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:04.415798 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:28:04.483217 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:28:04.606127 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:28:04.608108 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:28:04.657716 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:28:04.664417 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:04.677914 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:04.686677 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:04.696053 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:04.701771 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:28:04.710827 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:04.729867 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:04.751649 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:04.763489 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:04.779824 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:04.790205 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:28:04.838517 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:04.878597 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:04.878590 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:04.897984 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:04.902830 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:28:04.913365 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:28:04.966624 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:05.041538 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:05.055040 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:05.069530 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:05.075120 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:05.076433 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:05.080056 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:05.193078 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:28:05.201711 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:05.272231 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:28:05.288802 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:05.289971 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:05.294448 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:05.296206 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:05.296725 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:05.301149 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:05.345722 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:28:05.563447 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:05.575189 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:05.595584 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:05.618928 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:05.647325 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:05.674949 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:05.678637 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:05.681381 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:05.692301 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:05.704530 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:05.718151 20964 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.5.107.71:39195, user_credentials={real_user=slave}} blocked reactor thread for 110284us
W20240502 14:28:05.720518 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:05.724123 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:05.729447 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240502 14:28:05.729527 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:05.730563 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:28:05.750504 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 14:28:05.753690 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:28:05.758307 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:05.762567 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 14:28:05.836479 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 14:28:05.852547 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:28:05.952519 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:28:05.980803 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:05.994810 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:06.022109 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:28:06.037325 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:06.064921 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:28:06.083643 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:06.084733 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:06.122017 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:06.139879 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:06.149989 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:06.165971 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 14:28:06.170547 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:06.179726 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:06.226611 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:06.231382 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:06.268136 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:06.283895 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:28:06.287225 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:06.301894 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:28:06.415419 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:28:06.455034 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:28:06.543267 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:06.653367 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:06.677664 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:28:06.684326 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:06.700824 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:06.707147 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:06.717023 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:28:06.732478 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:28:06.735185 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:28:06.743820 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:06.776603 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:06.781019 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:28:06.799772 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:28:06.804567 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:06.845762 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:28:06.870417 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:06.884393 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:06.906867 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:06.912442 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:06.925316 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:06.935431 20562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 14:28:06.949646 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:07.014909 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:07.030793 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:28:07.042690 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:28:07.062479 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:28:07.118084 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:28:07.124363 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:07.145164 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:28:07.153131 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:07.157410 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:28:07.175246 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:07.184908 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:07.226490 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:07.226938 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:07.232770 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:07.350499 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:07.351039 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:07.379766 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:28:07.386029 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:28:07.387743 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:07.392725 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:07.393810 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:28:07.400753 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:07.425531 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:07.429884 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:07.620508 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:07.627784 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:07.628597 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:07.636606 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:07.684103 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:07.700157 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:07.705286 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:07.822376 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:07.834573 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:07.866748 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:28:07.871287 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:07.876154 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:07.883785 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:07.883951 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:07.885036 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:07.914968 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:28:08.087850 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:28:08.123951 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:08.133944 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:08.176677 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:08.189553 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:08.203581 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:08.222546 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:08.226938 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:28:08.241134 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:08.275593 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:08.290066 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:08.290758 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:08.299338 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:08.308224 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:08.318197 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:08.366664 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:08.387179 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:28:08.466104 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:28:08.491770 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:28:08.566892 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:28:08.585666 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:08.591745 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:08.610035 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:08.611882 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:08.619666 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:08.638216 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:08.689707 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:28:08.727325 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:08.728101 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:08.857915 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:28:08.857980 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:08.875478 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:08.888329 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:08.939559 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:08.941766 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:08.946004 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:28:08.984982 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:09.063225 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:09.083776 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:09.093977 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:09.108691 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:09.116143 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:09.116240 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:09.118960 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:09.134775 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:09.138785 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:28:09.144418 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:28:09.199987 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:28:09.209347 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:28:09.284447 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:28:09.285768 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:28:09.295083 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:09.295579 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:28:09.303766 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:09.308003 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:28:09.313875 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:09.315129 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:28:09.316735 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:09.331008 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:09.339038 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:09.427471 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:28:09.450495 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:28:09.480444 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:28:09.563871 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:09.570626 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:09.593907 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:28:09.596776 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:09.598484 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:09.601070 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:28:09.612746 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:09.619576 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:09.638309 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:09.674422 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:09.679008 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:28:09.833036 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:28:09.896090 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:09.908229 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:09.912518 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:09.915951 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:09.933039 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:09.941092 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:09.949002 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:09.986778 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:10.013038 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:10.044740 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:10.058971 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:10.090807 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:10.118201 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:10.182860 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:10.233239 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:28:10.313771 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:10.317696 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:10.322398 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:10.332173 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:10.350045 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:10.352000 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:10.445855 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:28:10.480562 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:28:10.499938 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:10.509712 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:10.518088 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:10.520819 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:10.526154 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:10.531577 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:10.639508 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:28:10.669308 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:28:10.672206 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:10.727965 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:28:10.773615 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 14:28:10.796680 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:10.801918 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:10.809336 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:10.810285 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:10.814965 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:10.816247 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:10.817129 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 87 similar messages]
I20240502 14:28:10.847954 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:10.849169 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 14:28:10.851958 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:28:10.963919 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 14:28:10.968829 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:28:11.037299 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:28:11.053941 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:11.091478 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:11.093865 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:11.111320 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:11.111704 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:11.144475 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:11.154284 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:28:11.272815 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:28:11.282344 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:11.284083 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:11.285863 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:11.294255 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:11.303478 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:11.308799 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:11.331328 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:11.351276 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:28:11.380298 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:11.452991 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:28:11.498248 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:11.500928 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:11.509960 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:11.523324 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:11.534878 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:11.535506 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:11.595402 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:28:11.657212 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:28:11.695466 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:11.696336 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:28:11.783178 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:28:11.793502 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:28:11.800827 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:28:11.816556 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:11.827661 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:28:11.831343 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:28:11.881677 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:28:11.904126 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:28:11.954032 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:28:11.990526 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:11.997896 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:28:12.053161 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:12.075585 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:12.082351 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:12.090143 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:12.097290 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:28:12.100824 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:12.105290 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:12.108088 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:12.115700 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:28:12.122825 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:12.155278 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:12.163776 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:12.189204 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:12.193647 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 14:28:12.236696 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:12.277935 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:28:12.291559 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:28:12.419592 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:12.421345 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:12.426761 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:12.433095 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:12.450381 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:12.473562 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:12.481534 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:12.682011 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:12.698539 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:12.698709 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:12.700693 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:12.714720 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:12.716164 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:12.729981 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:12.868732 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:28:12.931728 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:12.943059 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:12.946383 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:28:12.949007 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:12.972092 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:12.973800 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:12.993964 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:13.015462 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:13.046655 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:28:13.052868 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:28:13.079416 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:13.099727 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:13.125605 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:28:13.137367 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:13.142660 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:13.158418 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:13.173458 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:13.187506 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:28:13.207080 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:13.308773 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:28:13.358991 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:13.391968 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:28:13.392128 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:13.404795 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:13.434466 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:13.444530 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:13.451151 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:13.454686 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:13.619722 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:13.621206 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:13.626403 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:13.627398 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:13.630872 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:13.635897 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:13.643033 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:13.719748 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:28:13.809398 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:28:13.905405 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:28:13.939991 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:13.962594 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:13.966775 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:13.982825 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:14.006942 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:14.029249 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:28:14.042690 20964 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.5.107.71:39195, user_credentials={real_user=slave}} blocked reactor thread for 79070.7us
W20240502 14:28:14.043936 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:14.049057 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:14.086637 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:28:14.106954 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:28:14.115141 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:14.167769 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:28:14.181301 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:28:14.189591 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:28:14.224526 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:14.236004 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:14.275969 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:14.290845 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:28:14.296425 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:14.297363 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:14.315584 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:28:14.317000 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:14.336093 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:28:14.336871 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:28:14.356901 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:28:14.374815 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:28:14.409114 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:28:14.410816 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:28:14.454252 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:14.454936 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:14.458563 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:14.476297 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:14.483578 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:14.502087 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:28:14.515601 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:14.516278 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:14.592329 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:14.662031 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:14.669457 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:14.673609 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:14.711666 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:14.736371 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:14.737219 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:14.811741 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:28:14.840698 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:28:14.955966 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:14.974175 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:14.983640 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:14.990423 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:14.999817 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:15.003450 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:15.013095 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:15.138196 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:15.143935 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:15.152026 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:15.176491 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:15.181123 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:28:15.188313 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:15.204195 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:15.211599 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:15.293035 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:28:15.399425 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:15.411331 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:15.441074 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:15.444365 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:15.452973 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:15.453316 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:28:15.454861 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:15.508705 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:15.544195 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:28:15.544512 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:28:15.561672 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:15.636050 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:15.648983 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:15.665356 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:15.672287 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:15.687829 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:15.707644 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:15.775832 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 14:28:15.779692 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:28:15.848683 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:28:15.865650 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 94 similar messages]
W20240502 14:28:15.947666 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 14:28:15.951334 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:15.954159 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:15.967765 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:15.995872 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:16.004726 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:16.017902 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:16.028093 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:16.028553 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 14:28:16.045372 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:16.155330 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:28:16.160539 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:28:16.199404 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:16.204870 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:16.231047 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:16.247378 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:16.262821 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:16.265313 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:28:16.273880 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:28:16.275893 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:16.372967 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:28:16.410986 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:16.417971 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:16.445531 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:16.454013 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:16.463057 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:16.475229 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:16.556427 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:16.573655 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:28:16.578810 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:28:16.589902 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:16.663491 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:16.670053 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:16.685247 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:16.688321 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:28:16.688681 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:28:16.690934 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:16.690930 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:16.721828 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:16.776880 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:28:16.780751 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:28:16.789264 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:28:16.850311 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:28:16.854457 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:28:16.857111 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:28:16.870436 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:28:16.879173 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:28:16.913867 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:28:16.932523 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:16.942914 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:16.950392 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:16.952308 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:16.967964 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:16.977535 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:16.982877 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:28:17.131086 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:17.173314 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:17.193708 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:28:17.210016 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:17.234333 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:17.275074 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:17.303035 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:17.309149 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:17.328836 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:17.366098 20562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:28:17.386394 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:28:17.458160 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:17.464584 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:17.469872 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:28:17.488234 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:17.513789 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:17.520699 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:17.546854 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:17.580281 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:17.622061 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:17.713085 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:17.716049 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:17.721729 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:17.726555 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:17.728163 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:28:17.734819 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:17.751694 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:17.762482 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:28:17.770954 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:28:17.920183 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:28:17.990568 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:18.008065 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:18.042841 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:18.057516 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:18.058774 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:18.090333 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:18.109124 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:18.110209 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:28:18.129635 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:18.145829 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:18.151319 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:18.156553 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:18.190407 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:18.231061 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:18.243017 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:18.256223 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:28:18.413007 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:18.417245 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:18.428525 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:18.429388 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:18.455693 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:18.487172 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:18.519440 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:28:18.525727 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:18.533087 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:28:18.541884 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:18.677386 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:18.688182 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:18.690838 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:18.694841 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:28:18.701627 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:18.706589 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:18.755744 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:18.786857 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:28:18.893913 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:28:18.957813 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:18.957787 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:28:18.969969 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:18.969969 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:19.001583 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:19.004343 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:19.010946 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:28:19.011354 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:28:19.019951 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:19.042241 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:19.053428 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:19.121130 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:19.130992 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:28:19.131686 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:19.158301 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:19.162102 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:19.169734 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:19.198240 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:19.229219 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:28:19.255216 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:28:19.255980 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:28:19.263931 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:28:19.278205 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:28:19.371311 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:28:19.397998 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:28:19.428349 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:19.432750 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:19.443531 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:19.458011 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:19.512887 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:19.525430 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:19.529765 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:19.536468 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:19.575649 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:28:19.575788 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:28:19.606283 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:19.617992 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:19.626914 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:19.628556 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:28:19.634052 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:19.644690 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:19.648170 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:19.808434 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:28:19.867383 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:28:19.891928 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:28:19.959661 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:28:19.998701 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:20.035737 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:20.077314 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:20.084101 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:20.124083 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:20.130486 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:20.153522 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:28:20.165251 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:20.227787 18549 maintenance_manager.cc:419] P 0283476f38234a96bfe72594576461be: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033397
I20240502 14:28:20.243300 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:20.330279 18547 maintenance_manager.cc:643] P 0283476f38234a96bfe72594576461be: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.101s user 0.051s sys 0.008s Metrics: {"bytes_written":20470,"cfile_init":1,"dirs.queue_time_us":12504,"dirs.run_cpu_time_us":287,"dirs.run_wall_time_us":2035,"drs_written":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1410,"lbm_writes_lt_1ms":28,"peak_mem_usage":0,"rows_written":43,"thread_start_us":22677,"threads_started":2,"wal-append.queue_time_us":10585}
I20240502 14:28:20.331647 18549 maintenance_manager.cc:419] P 0283476f38234a96bfe72594576461be: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7341 bytes on disk
I20240502 14:28:20.332597 18547 maintenance_manager.cc:643] P 0283476f38234a96bfe72594576461be: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240502 14:28:20.367009 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:20.388366 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:20.404420 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:28:20.415244 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:20.429312 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:20.444008 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:20.459081 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:20.465652 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:20.485675 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:20.485811 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:28:20.653649 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:20.663345 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:20.667538 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:20.681710 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:20.691642 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:20.695667 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:20.779353 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 76 similar messages]
W20240502 14:28:20.780267 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:28:20.831519 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:20.859413 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:20.866010 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 90 similar messages]
I20240502 14:28:20.874145 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:20.876130 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:20.877027 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:20.885799 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:20.892824 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:20.893527 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:20.953256 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:28:20.962615 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 14:28:21.047505 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:28:21.069166 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:28:21.077299 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:28:21.125360 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:21.136687 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:21.174410 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:21.186153 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 14:28:21.189952 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:21.199606 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:21.203081 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:21.252717 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:21.309769 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:28:21.317410 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:21.318368 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:21.330004 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:28:21.346030 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:21.354015 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:28:21.363679 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:21.369469 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:21.417429 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:21.431200 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:28:21.442113 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:21.550655 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:28:21.561098 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:21.568295 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:21.584456 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:21.592674 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:21.609592 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:21.646950 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:21.676143 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:28:21.690153 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:28:21.727042 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:28:21.758472 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:28:21.806903 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:28:21.828402 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:28:21.847182 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:21.850697 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:28:21.854877 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:21.861541 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:28:21.862848 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:21.869920 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:21.887521 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:21.901482 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:21.902678 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:21.906262 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:22.003379 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:28:22.069936 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:28:22.098268 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:28:22.109974 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:22.129479 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:22.133872 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:28:22.208735 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:22.210495 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:22.231509 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:22.234426 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:28:22.266250 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:22.271731 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:22.284093 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:28:22.301676 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:22.358027 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:22.370244 20562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 18 similar messages]
I20240502 14:28:22.396596 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:22.401553 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:22.403191 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:28:22.419268 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:22.429471 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:22.430418 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:22.471156 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:28:22.717082 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:22.730203 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:22.742223 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:22.746874 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:28:22.768102 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:22.775038 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:22.794420 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:22.795647 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:22.855432 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:22.868767 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:22.870329 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:22.897557 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:22.905627 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:22.930415 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:22.950827 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:28:22.957233 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:23.143333 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:28:23.367039 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:28:23.377980 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:23.411617 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:23.417738 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:23.435493 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:23.439878 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:23.441582 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:23.477869 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:23.489897 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:23.502679 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:23.538990 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:23.565084 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:28:23.565464 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:23.565838 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:28:23.581441 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:28:23.586177 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:23.596304 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:28:23.605306 20964 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.5.107.71:39195, user_credentials={real_user=slave}} blocked reactor thread for 61811.7us
W20240502 14:28:23.608799 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:23.618468 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:23.630789 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:28:23.728999 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:28:23.788780 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:28:23.854239 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:23.881335 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:23.892484 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:23.910569 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:23.916042 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:28:23.916191 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:23.918921 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:23.923877 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:23.924635 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:23.942898 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:23.947402 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:23.958802 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:24.010979 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:24.029259 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:28:24.040760 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:24.045296 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:24.091630 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:28:24.106870 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:28:24.115146 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:28:24.157155 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:28:24.252900 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:24.265540 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:24.273648 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:24.289196 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:24.295248 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:24.309345 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:24.322451 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:24.364560 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:28:24.446507 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:28:24.449928 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:24.483134 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:28:24.495054 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:28:24.525527 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:24.526903 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:24.532780 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:24.549638 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:24.559424 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:28:24.562456 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:24.577014 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:24.594213 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:28:24.619318 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:28:24.652120 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:28:24.702533 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:24.843932 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:24.863641 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:24.865483 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:24.893771 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:24.894635 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:28:24.917598 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:24.933045 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:24.934120 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:24.936339 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:24.939321 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:25.001914 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:25.021248 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:25.022251 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:28:25.027925 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:25.045914 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:25.132184 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:25.138947 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:28:25.146842 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:28:25.203765 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:25.214111 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:25.221186 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:25.245563 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:25.249599 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:25.252341 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:25.361325 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:28:25.458176 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:28:25.514730 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:25.552513 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:25.561151 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:25.572459 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:25.581363 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:25.581358 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:25.596058 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:25.617523 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:25.617509 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:25.639225 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:25.683503 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:25.710543 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:25.766155 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:25.799727 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:25.818717 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 14:28:25.850009 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:28:25.919646 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 14:28:25.920512 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:28:25.947162 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:25.947616 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:25.964748 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:28:25.966377 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:25.970146 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:25.979442 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:25.985314 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:26.054085 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:28:26.083124 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 14:28:26.084204 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:28:26.085551 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:28:26.114863 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:26.158452 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:26.189419 19032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 14:28:26.215634 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:26.216432 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:26.220161 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:26.230718 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:28:26.242861 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:26.252418 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:26.277268 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:26.333380 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:28:26.434007 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:28:26.435704 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:28:26.492601 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:28:26.516481 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:26.521425 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:26.530117 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:26.536700 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:28:26.557174 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:26.575773 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:26.592659 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:26.593806 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:26.606310 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:26.615797 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:26.629662 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:26.636938 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:28:26.653928 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:26.659862 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:26.695283 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:28:26.716349 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:26.725617 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:26.831014 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:28:26.948194 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:26.966383 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:26.984781 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:26.985154 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:26.989820 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:28:26.994614 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:27.000918 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:27.001960 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:28:27.003558 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:27.013521 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:27.016366 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:28:27.062394 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:28:27.081390 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:28:27.110766 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:27.114305 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:27.132108 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:28:27.143198 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:27.144313 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:27.156908 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:27.157510 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:28:27.166254 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:27.242717 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:28:27.332309 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:28:27.440793 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:27.445500 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:27.449041 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:27.452270 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:28:27.456171 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:27.457161 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:27.460202 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:27.543011 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:27.595549 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:27.634284 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:28:27.635427 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:28:27.737028 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:27.737727 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:28:27.764537 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:27.768191 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:27.804013 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:27.815421 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:28:27.832175 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:27.851615 18748 maintenance_manager.cc:419] P 2db304b538e14857b7b1cc272d0db94b: Scheduling FlushMRSOp(d28db16c9f7e405b8866f0e6c548e246): perf score=0.033347
I20240502 14:28:27.853682 18880 maintenance_manager.cc:419] P fec04c50dc5b49878ffa3c3873df32e4: Scheduling FlushMRSOp(9580f3fdf7084cc0923b1b16bfedd450): perf score=0.033347
I20240502 14:28:27.865502 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:27.880260 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:28:27.885097 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:27.891463 19031 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.72:35871, user_credentials={real_user=slave}} blocked reactor thread for 43721.6us
I20240502 14:28:27.895079 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:27.936925 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:27.938724 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:27.970726 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:28.009280 19012 maintenance_manager.cc:419] P c902805060df4df9b5599d0e94b2dc06: Scheduling FlushMRSOp(cf2a6426ae6c438f942d39530da6e3e3): perf score=0.033389
I20240502 14:28:28.026726 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:28.048094 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:28.068915 18638 maintenance_manager.cc:643] P 2db304b538e14857b7b1cc272d0db94b: FlushMRSOp(d28db16c9f7e405b8866f0e6c548e246) complete. Timing: real 0.206s user 0.069s sys 0.014s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":25783,"dirs.queue_time_us":7735,"dirs.run_cpu_time_us":281,"dirs.run_wall_time_us":8376,"drs_written":1,"lbm_read_time_us":230,"lbm_reads_lt_1ms":4,"lbm_write_time_us":845,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":3840,"thread_start_us":51921,"threads_started":3,"wal-append.queue_time_us":13186}
I20240502 14:28:28.070672 18748 maintenance_manager.cc:419] P 2db304b538e14857b7b1cc272d0db94b: Scheduling UndoDeltaBlockGCOp(d28db16c9f7e405b8866f0e6c548e246): 4923 bytes on disk
I20240502 14:28:28.071667 18638 maintenance_manager.cc:643] P 2db304b538e14857b7b1cc272d0db94b: UndoDeltaBlockGCOp(d28db16c9f7e405b8866f0e6c548e246) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
W20240502 14:28:28.267454 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:28:28.271827 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:28.292286 18901 maintenance_manager.cc:643] P c902805060df4df9b5599d0e94b2dc06: FlushMRSOp(cf2a6426ae6c438f942d39530da6e3e3) complete. Timing: real 0.282s user 0.076s sys 0.020s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":16175,"dirs.queue_time_us":10797,"dirs.run_cpu_time_us":289,"dirs.run_wall_time_us":6496,"drs_written":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4,"lbm_write_time_us":25557,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":48757,"threads_started":3,"wal-append.queue_time_us":7563}
W20240502 14:28:28.294818 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:28:28.298473 19012 maintenance_manager.cc:419] P c902805060df4df9b5599d0e94b2dc06: Scheduling UndoDeltaBlockGCOp(cf2a6426ae6c438f942d39530da6e3e3): 4840 bytes on disk
I20240502 14:28:28.300140 18901 maintenance_manager.cc:643] P c902805060df4df9b5599d0e94b2dc06: UndoDeltaBlockGCOp(cf2a6426ae6c438f942d39530da6e3e3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240502 14:28:28.304746 18770 maintenance_manager.cc:643] P fec04c50dc5b49878ffa3c3873df32e4: FlushMRSOp(9580f3fdf7084cc0923b1b16bfedd450) complete. Timing: real 0.450s user 0.094s sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":46287,"dirs.queue_time_us":6831,"dirs.run_cpu_time_us":299,"dirs.run_wall_time_us":7025,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":12102,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":56548,"threads_started":3,"wal-append.queue_time_us":9962}
I20240502 14:28:28.306450 18880 maintenance_manager.cc:419] P fec04c50dc5b49878ffa3c3873df32e4: Scheduling UndoDeltaBlockGCOp(9580f3fdf7084cc0923b1b16bfedd450): 4586 bytes on disk
I20240502 14:28:28.314447 18770 maintenance_manager.cc:643] P fec04c50dc5b49878ffa3c3873df32e4: UndoDeltaBlockGCOp(9580f3fdf7084cc0923b1b16bfedd450) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":200,"lbm_reads_lt_1ms":4}
W20240502 14:28:28.380522 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:28:28.393326 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:28:28.475936 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:28.559141 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:28:28.639583 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:28.647701 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:28:28.653054 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:28.668604 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:28.702328 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:28.706969 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:28.774260 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:28.782516 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:28:28.807698 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:28.862809 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:28:28.880470 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:28.917814 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:28:28.930475 19143 maintenance_manager.cc:419] P 007b2e79dc424a6f80141180f072f2d8: Scheduling FlushMRSOp(8937c809610542d88fdde3daf002c890): perf score=0.033354
I20240502 14:28:29.069016 18880 maintenance_manager.cc:419] P fec04c50dc5b49878ffa3c3873df32e4: Scheduling FlushMRSOp(6acde0e5509340fbb52bca18645a77b7): perf score=0.033391
I20240502 14:28:29.076534 18748 maintenance_manager.cc:419] P 2db304b538e14857b7b1cc272d0db94b: Scheduling FlushMRSOp(0ffb6e582ef04fb38dd273c13febac58): perf score=0.033398
W20240502 14:28:29.112188 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:28:29.112468 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:28:29.122640 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:28:29.128119 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:29.134159 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:29.154788 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:29.158672 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:28:29.159591 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:29.184484 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:29.197660 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:28:29.207348 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:29.261792 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:29.277917 18770 maintenance_manager.cc:643] P fec04c50dc5b49878ffa3c3873df32e4: FlushMRSOp(6acde0e5509340fbb52bca18645a77b7) complete. Timing: real 0.208s user 0.038s sys 0.017s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":22298,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":4403,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4243,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":35369,"threads_started":2,"wal-append.queue_time_us":12560}
I20240502 14:28:29.279491 18880 maintenance_manager.cc:419] P fec04c50dc5b49878ffa3c3873df32e4: Scheduling UndoDeltaBlockGCOp(6acde0e5509340fbb52bca18645a77b7): 4586 bytes on disk
I20240502 14:28:29.284837 18770 maintenance_manager.cc:643] P fec04c50dc5b49878ffa3c3873df32e4: UndoDeltaBlockGCOp(6acde0e5509340fbb52bca18645a77b7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
W20240502 14:28:29.292438 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:29.294451 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:28:29.345044 18638 maintenance_manager.cc:643] P 2db304b538e14857b7b1cc272d0db94b: FlushMRSOp(0ffb6e582ef04fb38dd273c13febac58) complete. Timing: real 0.268s user 0.047s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":5196,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":3715,"drs_written":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4,"lbm_write_time_us":12156,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":23773,"threads_started":2,"wal-append.queue_time_us":6325}
I20240502 14:28:29.348697 18748 maintenance_manager.cc:419] P 2db304b538e14857b7b1cc272d0db94b: Scheduling UndoDeltaBlockGCOp(0ffb6e582ef04fb38dd273c13febac58): 4840 bytes on disk
I20240502 14:28:29.352802 18638 maintenance_manager.cc:643] P 2db304b538e14857b7b1cc272d0db94b: UndoDeltaBlockGCOp(0ffb6e582ef04fb38dd273c13febac58) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
W20240502 14:28:29.419191 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:28:29.444604 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:28:29.449892 19033 maintenance_manager.cc:643] P 007b2e79dc424a6f80141180f072f2d8: FlushMRSOp(8937c809610542d88fdde3daf002c890) complete. Timing: real 0.518s user 0.123s sys 0.037s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":13026,"dirs.queue_time_us":58213,"dirs.run_cpu_time_us":260,"dirs.run_wall_time_us":3100,"drs_written":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4,"lbm_write_time_us":38052,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":78148,"threads_started":3,"wal-append.queue_time_us":3818}
I20240502 14:28:29.451805 19143 maintenance_manager.cc:419] P 007b2e79dc424a6f80141180f072f2d8: Scheduling UndoDeltaBlockGCOp(8937c809610542d88fdde3daf002c890): 4925 bytes on disk
I20240502 14:28:29.452785 19033 maintenance_manager.cc:643] P 007b2e79dc424a6f80141180f072f2d8: UndoDeltaBlockGCOp(8937c809610542d88fdde3daf002c890) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240502 14:28:29.454351 19143 maintenance_manager.cc:419] P 007b2e79dc424a6f80141180f072f2d8: Scheduling FlushMRSOp(6acde0e5509340fbb52bca18645a77b7): perf score=0.033496
W20240502 14:28:29.503494 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:28:29.526595 19033 maintenance_manager.cc:643] P 007b2e79dc424a6f80141180f072f2d8: FlushMRSOp(6acde0e5509340fbb52bca18645a77b7) complete. Timing: real 0.072s user 0.039s sys 0.024s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":61,"dirs.run_cpu_time_us":260,"dirs.run_wall_time_us":4590,"drs_written":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2954,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6672,"threads_started":1,"wal-append.queue_time_us":6490}
I20240502 14:28:29.527956 19143 maintenance_manager.cc:419] P 007b2e79dc424a6f80141180f072f2d8: Scheduling UndoDeltaBlockGCOp(6acde0e5509340fbb52bca18645a77b7): 4586 bytes on disk
I20240502 14:28:29.528837 19033 maintenance_manager.cc:643] P 007b2e79dc424a6f80141180f072f2d8: UndoDeltaBlockGCOp(6acde0e5509340fbb52bca18645a77b7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":126,"lbm_reads_lt_1ms":4}
W20240502 14:28:29.635134 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:28:29.666574 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:28:29.745747 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:29.754006 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:29.756193 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:28:29.780969 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:29.782584 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:29.792694 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:29.820310 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:29.820911 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:29.845579 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:29.865326 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:28:29.865670 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:28:29.925809 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:28:30.021816 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:30.027469 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:30.094794 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:30.098582 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:30.110733 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:30.162647 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:30.208631 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:28:30.219378 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:28:30.237962 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:30.242131 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:28:30.337549 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:30.363895 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:30.377790 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:28:30.394691 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:30.396229 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:30.415076 18768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.72:35871, user_credentials={real_user=slave}} blocked reactor thread for 47422.9us
I20240502 14:28:30.421782 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:30.441651 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:30.443533 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:30.629609 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:28:30.634744 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:30.737653 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:30.743450 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:30.769170 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:30.812811 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:30.812923 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:30.820458 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:30.827675 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:30.859689 18748 maintenance_manager.cc:419] P 2db304b538e14857b7b1cc272d0db94b: Scheduling FlushMRSOp(96662a3d9e1b4f03b05464ddd06b93d6): perf score=0.033336
I20240502 14:28:30.883667 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:30.887373 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 14:28:30.921695 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 88 similar messages]
I20240502 14:28:30.922899 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:30.923528 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:28:30.927323 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:30.944219 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:30.944792 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:30.950027 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:28:30.953644 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:30.987088 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:28:31.024026 18638 maintenance_manager.cc:643] P 2db304b538e14857b7b1cc272d0db94b: FlushMRSOp(96662a3d9e1b4f03b05464ddd06b93d6) complete. Timing: real 0.164s user 0.074s sys 0.028s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":3918,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":3513,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":8387,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":635,"thread_start_us":42612,"threads_started":2,"wal-append.queue_time_us":38831}
I20240502 14:28:31.025576 18748 maintenance_manager.cc:419] P 2db304b538e14857b7b1cc272d0db94b: Scheduling UndoDeltaBlockGCOp(96662a3d9e1b4f03b05464ddd06b93d6): 4586 bytes on disk
I20240502 14:28:31.026330 18638 maintenance_manager.cc:643] P 2db304b538e14857b7b1cc272d0db94b: UndoDeltaBlockGCOp(96662a3d9e1b4f03b05464ddd06b93d6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240502 14:28:31.044591 18880 maintenance_manager.cc:419] P fec04c50dc5b49878ffa3c3873df32e4: Scheduling FlushMRSOp(0d8667d91fd64809b635362c4a8fa33f): perf score=0.033399
W20240502 14:28:31.045393 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:31.065152 19012 maintenance_manager.cc:419] P c902805060df4df9b5599d0e94b2dc06: Scheduling FlushMRSOp(96662a3d9e1b4f03b05464ddd06b93d6): perf score=0.033396
W20240502 14:28:31.072932 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:28:31.115025 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 14:28:31.138485 18901 maintenance_manager.cc:643] P c902805060df4df9b5599d0e94b2dc06: FlushMRSOp(96662a3d9e1b4f03b05464ddd06b93d6) complete. Timing: real 0.073s user 0.039s sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":4387,"dirs.run_cpu_time_us":245,"dirs.run_wall_time_us":6120,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":852,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":10164,"threads_started":2,"wal-append.queue_time_us":5460}
I20240502 14:28:31.139993 19012 maintenance_manager.cc:419] P c902805060df4df9b5599d0e94b2dc06: Scheduling UndoDeltaBlockGCOp(96662a3d9e1b4f03b05464ddd06b93d6): 4586 bytes on disk
I20240502 14:28:31.140677 18901 maintenance_manager.cc:643] P c902805060df4df9b5599d0e94b2dc06: UndoDeltaBlockGCOp(96662a3d9e1b4f03b05464ddd06b93d6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240502 14:28:31.162457 18770 maintenance_manager.cc:643] P fec04c50dc5b49878ffa3c3873df32e4: FlushMRSOp(0d8667d91fd64809b635362c4a8fa33f) complete. Timing: real 0.115s user 0.072s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":3166,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":7941,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":871,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":3530,"threads_started":2,"wal-append.queue_time_us":341}
I20240502 14:28:31.163903 18880 maintenance_manager.cc:419] P fec04c50dc5b49878ffa3c3873df32e4: Scheduling UndoDeltaBlockGCOp(0d8667d91fd64809b635362c4a8fa33f): 4925 bytes on disk
I20240502 14:28:31.164475 18770 maintenance_manager.cc:643] P fec04c50dc5b49878ffa3c3873df32e4: UndoDeltaBlockGCOp(0d8667d91fd64809b635362c4a8fa33f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240502 14:28:31.255851 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:31.264312 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:31.306047 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:31.314891 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:31.327836 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:28:31.332962 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:31.351945 19032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 14:28:31.352454 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:28:31.356482 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:31.423297 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:28:31.426460 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:31.459565 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:28:31.460829 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:31.461923 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:31.468390 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:31.471880 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:31.482525 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:31.502337 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:28:31.504921 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:31.535907 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:31.573352 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:28:31.612207 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:28:31.622982 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:28:31.719026 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:28:31.766247 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:31.777005 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:31.784487 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:28:31.784628 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:31.785166 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:31.794306 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:28:31.797855 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:31.830137 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:31.894927 19012 maintenance_manager.cc:419] P c902805060df4df9b5599d0e94b2dc06: Scheduling FlushMRSOp(ecb8a9b04ba34f779e2227e77e6f764c): perf score=0.033352
W20240502 14:28:31.900030 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:31.918174 18880 maintenance_manager.cc:419] P fec04c50dc5b49878ffa3c3873df32e4: Scheduling FlushMRSOp(332e4b23751b422ab7bd17b80c93701d): perf score=0.033355
W20240502 14:28:31.953065 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:31.979736 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:28:31.988507 18770 maintenance_manager.cc:643] P fec04c50dc5b49878ffa3c3873df32e4: FlushMRSOp(332e4b23751b422ab7bd17b80c93701d) complete. Timing: real 0.067s user 0.046s sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6424,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":11162,"drs_written":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4,"lbm_write_time_us":846,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":17688,"threads_started":2,"wal-append.queue_time_us":11211}
I20240502 14:28:31.990092 18880 maintenance_manager.cc:419] P fec04c50dc5b49878ffa3c3873df32e4: Scheduling UndoDeltaBlockGCOp(332e4b23751b422ab7bd17b80c93701d): 4586 bytes on disk
I20240502 14:28:31.990955 18901 maintenance_manager.cc:643] P c902805060df4df9b5599d0e94b2dc06: FlushMRSOp(ecb8a9b04ba34f779e2227e77e6f764c) complete. Timing: real 0.096s user 0.067s sys 0.020s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":16647,"dirs.run_cpu_time_us":158,"dirs.run_wall_time_us":1618,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":842,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":19054,"threads_started":2,"wal-append.queue_time_us":2504}
I20240502 14:28:31.991787 18770 maintenance_manager.cc:643] P fec04c50dc5b49878ffa3c3873df32e4: UndoDeltaBlockGCOp(332e4b23751b422ab7bd17b80c93701d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240502 14:28:31.992355 19012 maintenance_manager.cc:419] P c902805060df4df9b5599d0e94b2dc06: Scheduling FlushMRSOp(332e4b23751b422ab7bd17b80c93701d): perf score=0.033367
W20240502 14:28:31.998044 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:28:32.030567 18901 maintenance_manager.cc:643] P c902805060df4df9b5599d0e94b2dc06: FlushMRSOp(332e4b23751b422ab7bd17b80c93701d) complete. Timing: real 0.038s user 0.026s sys 0.009s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1319,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":729,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":2304,"thread_start_us":313,"threads_started":1,"wal-append.queue_time_us":423}
I20240502 14:28:32.031872 18748 maintenance_manager.cc:419] P 2db304b538e14857b7b1cc272d0db94b: Scheduling FlushMRSOp(bad03a4ec8494938839d7d6eef8c684e): perf score=0.033382
I20240502 14:28:32.032029 19012 maintenance_manager.cc:419] P c902805060df4df9b5599d0e94b2dc06: Scheduling UndoDeltaBlockGCOp(ecb8a9b04ba34f779e2227e77e6f764c): 4925 bytes on disk
I20240502 14:28:32.041875 18901 maintenance_manager.cc:643] P c902805060df4df9b5599d0e94b2dc06: UndoDeltaBlockGCOp(ecb8a9b04ba34f779e2227e77e6f764c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20240502 14:28:32.042235 19143 maintenance_manager.cc:419] P 007b2e79dc424a6f80141180f072f2d8: Scheduling FlushMRSOp(bad03a4ec8494938839d7d6eef8c684e): perf score=0.033392
I20240502 14:28:32.043522 19012 maintenance_manager.cc:419] P c902805060df4df9b5599d0e94b2dc06: Scheduling UndoDeltaBlockGCOp(332e4b23751b422ab7bd17b80c93701d): 4586 bytes on disk
I20240502 14:28:32.046978 18901 maintenance_manager.cc:643] P c902805060df4df9b5599d0e94b2dc06: UndoDeltaBlockGCOp(332e4b23751b422ab7bd17b80c93701d) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":173,"lbm_reads_lt_1ms":4}
I20240502 14:28:32.152732 19033 maintenance_manager.cc:643] P 007b2e79dc424a6f80141180f072f2d8: FlushMRSOp(bad03a4ec8494938839d7d6eef8c684e) complete. Timing: real 0.110s user 0.045s sys 0.009s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":390,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":3224,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1202,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":3482,"threads_started":2,"wal-append.queue_time_us":346}
I20240502 14:28:32.154435 19143 maintenance_manager.cc:419] P 007b2e79dc424a6f80141180f072f2d8: Scheduling UndoDeltaBlockGCOp(bad03a4ec8494938839d7d6eef8c684e): 4840 bytes on disk
I20240502 14:28:32.155179 19033 maintenance_manager.cc:643] P 007b2e79dc424a6f80141180f072f2d8: UndoDeltaBlockGCOp(bad03a4ec8494938839d7d6eef8c684e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
W20240502 14:28:32.163558 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:28:32.169903 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:28:32.177595 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:32.209815 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:32.212776 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:32.226109 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:32.232226 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:32.260003 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:32.260900 18638 maintenance_manager.cc:643] P 2db304b538e14857b7b1cc272d0db94b: FlushMRSOp(bad03a4ec8494938839d7d6eef8c684e) complete. Timing: real 0.229s user 0.050s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":14164,"dirs.run_cpu_time_us":222,"dirs.run_wall_time_us":4838,"drs_written":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4,"lbm_write_time_us":11731,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":94695,"threads_started":2,"wal-append.queue_time_us":79086}
I20240502 14:28:32.263813 18748 maintenance_manager.cc:419] P 2db304b538e14857b7b1cc272d0db94b: Scheduling UndoDeltaBlockGCOp(bad03a4ec8494938839d7d6eef8c684e): 4840 bytes on disk
I20240502 14:28:32.264654 18638 maintenance_manager.cc:643] P 2db304b538e14857b7b1cc272d0db94b: UndoDeltaBlockGCOp(bad03a4ec8494938839d7d6eef8c684e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4}
I20240502 14:28:32.341521 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:32.345448 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:28:32.347720 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:32.360368 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:32.367070 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:28:32.380240 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:28:32.389456 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:32.398283 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:32.414250 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:32.457044 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:32.481935 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:28:32.500243 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:32.539331 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:28:32.552845 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:28:32.623745 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:28:32.651693 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:32.665805 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:32.675580 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:32.682575 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:32.686177 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:32.690594 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:32.751075 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:32.764442 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:28:32.765311 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:28:32.915812 19143 maintenance_manager.cc:419] P 007b2e79dc424a6f80141180f072f2d8: Scheduling FlushMRSOp(097eb705c9e24740a5be61bf3354f3d8): perf score=0.033360
W20240502 14:28:32.961845 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:32.997103 18880 maintenance_manager.cc:419] P fec04c50dc5b49878ffa3c3873df32e4: Scheduling FlushMRSOp(414e03d597934da3a6a74d786aeab134): perf score=0.033383
W20240502 14:28:33.007885 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:28:33.053109 19012 maintenance_manager.cc:419] P c902805060df4df9b5599d0e94b2dc06: Scheduling FlushMRSOp(b13bcd6e693f4c7687a1aabdafbb83c6): perf score=0.033399
I20240502 14:28:33.082924 19033 maintenance_manager.cc:643] P 007b2e79dc424a6f80141180f072f2d8: FlushMRSOp(097eb705c9e24740a5be61bf3354f3d8) complete. Timing: real 0.167s user 0.074s sys 0.021s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6384,"dirs.run_cpu_time_us":237,"dirs.run_wall_time_us":9606,"drs_written":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1098,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":13105,"threads_started":2,"wal-append.queue_time_us":6467}
I20240502 14:28:33.087471 19143 maintenance_manager.cc:419] P 007b2e79dc424a6f80141180f072f2d8: Scheduling UndoDeltaBlockGCOp(097eb705c9e24740a5be61bf3354f3d8): 4586 bytes on disk
I20240502 14:28:33.088120 19033 maintenance_manager.cc:643] P 007b2e79dc424a6f80141180f072f2d8: UndoDeltaBlockGCOp(097eb705c9e24740a5be61bf3354f3d8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4}
I20240502 14:28:33.089144 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:33.124222 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:33.130919 18770 maintenance_manager.cc:643] P fec04c50dc5b49878ffa3c3873df32e4: FlushMRSOp(414e03d597934da3a6a74d786aeab134) complete. Timing: real 0.134s user 0.061s sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":19898,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1533,"drs_written":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1013,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":26517,"threads_started":2,"wal-append.queue_time_us":6590}
I20240502 14:28:33.132833 18880 maintenance_manager.cc:419] P fec04c50dc5b49878ffa3c3873df32e4: Scheduling UndoDeltaBlockGCOp(414e03d597934da3a6a74d786aeab134): 4840 bytes on disk
I20240502 14:28:33.134167 18770 maintenance_manager.cc:643] P fec04c50dc5b49878ffa3c3873df32e4: UndoDeltaBlockGCOp(414e03d597934da3a6a74d786aeab134) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":133,"lbm_reads_lt_1ms":4}
I20240502 14:28:33.144302 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:33.166124 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:33.171782 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:33.197816 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:33.266171 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:33.278352 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:33.300709 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:28:33.301769 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:33.313007 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:33.327596 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:33.331027 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:33.360942 18901 maintenance_manager.cc:643] P c902805060df4df9b5599d0e94b2dc06: FlushMRSOp(b13bcd6e693f4c7687a1aabdafbb83c6) complete. Timing: real 0.308s user 0.058s sys 0.029s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6788,"dirs.run_cpu_time_us":233,"dirs.run_wall_time_us":8039,"drs_written":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4,"lbm_write_time_us":998,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":67363,"threads_started":2,"wal-append.queue_time_us":49144}
I20240502 14:28:33.364502 19012 maintenance_manager.cc:419] P c902805060df4df9b5599d0e94b2dc06: Scheduling UndoDeltaBlockGCOp(b13bcd6e693f4c7687a1aabdafbb83c6): 4925 bytes on disk
I20240502 14:28:33.365550 18901 maintenance_manager.cc:643] P c902805060df4df9b5599d0e94b2dc06: UndoDeltaBlockGCOp(b13bcd6e693f4c7687a1aabdafbb83c6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4}
W20240502 14:28:33.389470 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:33.426852 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:28:33.427733 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:28:33.432809 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:28:33.452617 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:28:33.490974 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:33.613351 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:33.625845 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:33.630375 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:33.644380 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:33.645310 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:33.657968 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:33.804502 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:28:33.863335 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:33.877776 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:33.884461 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:33.886209 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:33.892607 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:33.899349 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:28:33.903004 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:33.919832 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:33.923923 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:33.955192 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:28:33.971377 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:28:34.093595 19143 maintenance_manager.cc:419] P 007b2e79dc424a6f80141180f072f2d8: Scheduling FlushMRSOp(6a992d12aee7416096d1353533397386): perf score=0.033372
W20240502 14:28:34.108119 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:28:34.113349 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:28:34.118590 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:28:34.121865 19012 maintenance_manager.cc:419] P c902805060df4df9b5599d0e94b2dc06: Scheduling FlushMRSOp(b15d11209e45473e863766e5cd6e822f): perf score=0.033370
I20240502 14:28:34.129289 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:34.142684 18880 maintenance_manager.cc:419] P fec04c50dc5b49878ffa3c3873df32e4: Scheduling FlushMRSOp(67827c0262724897a05bf80df579bbf3): perf score=0.033379
I20240502 14:28:34.145891 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:34.164690 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:28:34.188091 19033 maintenance_manager.cc:643] P 007b2e79dc424a6f80141180f072f2d8: FlushMRSOp(6a992d12aee7416096d1353533397386) complete. Timing: real 0.094s user 0.057s sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":23791,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1471,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":10480,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":28572,"threads_started":2,"wal-append.queue_time_us":5579}
I20240502 14:28:34.189508 19143 maintenance_manager.cc:419] P 007b2e79dc424a6f80141180f072f2d8: Scheduling UndoDeltaBlockGCOp(6a992d12aee7416096d1353533397386): 4585 bytes on disk
I20240502 14:28:34.190291 19033 maintenance_manager.cc:643] P 007b2e79dc424a6f80141180f072f2d8: UndoDeltaBlockGCOp(6a992d12aee7416096d1353533397386) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":138,"lbm_reads_lt_1ms":4}
I20240502 14:28:34.215736 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:34.216737 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:34.251839 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:34.279606 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:34.301841 18901 maintenance_manager.cc:643] P c902805060df4df9b5599d0e94b2dc06: FlushMRSOp(b15d11209e45473e863766e5cd6e822f) complete. Timing: real 0.180s user 0.050s sys 0.028s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":36254,"dirs.run_cpu_time_us":254,"dirs.run_wall_time_us":25663,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":983,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":58400,"threads_started":2,"wal-append.queue_time_us":22035}
I20240502 14:28:34.303354 19012 maintenance_manager.cc:419] P c902805060df4df9b5599d0e94b2dc06: Scheduling FlushMRSOp(67827c0262724897a05bf80df579bbf3): perf score=0.033419
W20240502 14:28:34.307637 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:34.356626 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:34.360540 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:28:34.369583 18770 maintenance_manager.cc:643] P fec04c50dc5b49878ffa3c3873df32e4: FlushMRSOp(67827c0262724897a05bf80df579bbf3) complete. Timing: real 0.226s user 0.029s sys 0.032s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":18317,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":2904,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":943,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":26364,"threads_started":2,"wal-append.queue_time_us":8075}
I20240502 14:28:34.370950 18880 maintenance_manager.cc:419] P fec04c50dc5b49878ffa3c3873df32e4: Scheduling UndoDeltaBlockGCOp(67827c0262724897a05bf80df579bbf3): 4840 bytes on disk
I20240502 14:28:34.372140 18770 maintenance_manager.cc:643] P fec04c50dc5b49878ffa3c3873df32e4: UndoDeltaBlockGCOp(67827c0262724897a05bf80df579bbf3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20240502 14:28:34.390960 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:34.415627 18901 maintenance_manager.cc:643] P c902805060df4df9b5599d0e94b2dc06: FlushMRSOp(67827c0262724897a05bf80df579bbf3) complete. Timing: real 0.112s user 0.062s sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":34,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":1548,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2030,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":17898,"threads_started":1,"wal-append.queue_time_us":17857}
I20240502 14:28:34.416852 19012 maintenance_manager.cc:419] P c902805060df4df9b5599d0e94b2dc06: Scheduling UndoDeltaBlockGCOp(b15d11209e45473e863766e5cd6e822f): 4925 bytes on disk
I20240502 14:28:34.420051 18901 maintenance_manager.cc:643] P c902805060df4df9b5599d0e94b2dc06: UndoDeltaBlockGCOp(b15d11209e45473e863766e5cd6e822f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":258,"lbm_reads_lt_1ms":4}
I20240502 14:28:34.421322 19012 maintenance_manager.cc:419] P c902805060df4df9b5599d0e94b2dc06: Scheduling UndoDeltaBlockGCOp(67827c0262724897a05bf80df579bbf3): 4840 bytes on disk
I20240502 14:28:34.426824 18901 maintenance_manager.cc:643] P c902805060df4df9b5599d0e94b2dc06: UndoDeltaBlockGCOp(67827c0262724897a05bf80df579bbf3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20240502 14:28:34.452692 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:34.470397 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:34.500674 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:34.505662 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:28:34.507932 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:34.511857 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:34.513487 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:28:34.538240 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:28:34.562487 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:28:34.606819 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:28:34.619529 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:28:34.666116 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:28:34.696988 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:34.707423 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:34.726907 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:34.778092 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:34.780309 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:34.780663 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:34.915802 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:34.934757 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:34.938026 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:28:34.948086 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:34.953397 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:34.956632 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:34.958038 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:34.973222 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:35.015419 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:28:35.030261 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:28:35.030961 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:35.102803 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:28:35.125986 18880 maintenance_manager.cc:419] P fec04c50dc5b49878ffa3c3873df32e4: Scheduling FlushMRSOp(8a2bed29fcd64f7bbb04286404ef09de): perf score=0.033342
I20240502 14:28:35.181725 19012 maintenance_manager.cc:419] P c902805060df4df9b5599d0e94b2dc06: Scheduling FlushMRSOp(8a2bed29fcd64f7bbb04286404ef09de): perf score=0.033360
I20240502 14:28:35.186537 18770 maintenance_manager.cc:643] P fec04c50dc5b49878ffa3c3873df32e4: FlushMRSOp(8a2bed29fcd64f7bbb04286404ef09de) complete. Timing: real 0.060s user 0.055s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":346,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":1582,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":606,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":472,"threads_started":2,"wal-append.queue_time_us":301}
I20240502 14:28:35.188019 18880 maintenance_manager.cc:419] P fec04c50dc5b49878ffa3c3873df32e4: Scheduling UndoDeltaBlockGCOp(8a2bed29fcd64f7bbb04286404ef09de): 4586 bytes on disk
I20240502 14:28:35.188730 18770 maintenance_manager.cc:643] P fec04c50dc5b49878ffa3c3873df32e4: UndoDeltaBlockGCOp(8a2bed29fcd64f7bbb04286404ef09de) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":138,"lbm_reads_lt_1ms":4}
I20240502 14:28:35.196985 19143 maintenance_manager.cc:419] P 007b2e79dc424a6f80141180f072f2d8: Scheduling FlushMRSOp(e5fb3b049ba94af998791640bd0e31c5): perf score=0.033344
W20240502 14:28:35.201406 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:28:35.261759 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:28:35.277973 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:28:35.279381 18748 maintenance_manager.cc:419] P 2db304b538e14857b7b1cc272d0db94b: Scheduling FlushMRSOp(b315327141954b53a0f87c2d1dc14b9b): perf score=0.033383
I20240502 14:28:35.283910 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:35.289716 19033 maintenance_manager.cc:643] P 007b2e79dc424a6f80141180f072f2d8: FlushMRSOp(e5fb3b049ba94af998791640bd0e31c5) complete. Timing: real 0.092s user 0.049s sys 0.017s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":10373,"dirs.run_cpu_time_us":245,"dirs.run_wall_time_us":6681,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":950,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":11505,"threads_started":2,"wal-append.queue_time_us":1078}
I20240502 14:28:35.291221 19143 maintenance_manager.cc:419] P 007b2e79dc424a6f80141180f072f2d8: Scheduling UndoDeltaBlockGCOp(e5fb3b049ba94af998791640bd0e31c5): 4840 bytes on disk
I20240502 14:28:35.291831 19033 maintenance_manager.cc:643] P 007b2e79dc424a6f80141180f072f2d8: UndoDeltaBlockGCOp(e5fb3b049ba94af998791640bd0e31c5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
W20240502 14:28:35.294137 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:35.294669 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:35.295502 18901 maintenance_manager.cc:643] P c902805060df4df9b5599d0e94b2dc06: FlushMRSOp(8a2bed29fcd64f7bbb04286404ef09de) complete. Timing: real 0.113s user 0.068s sys 0.023s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":16674,"dirs.run_cpu_time_us":246,"dirs.run_wall_time_us":3931,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":988,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":1792,"thread_start_us":23016,"threads_started":2,"wal-append.queue_time_us":6350}
I20240502 14:28:35.298278 19012 maintenance_manager.cc:419] P c902805060df4df9b5599d0e94b2dc06: Scheduling UndoDeltaBlockGCOp(8a2bed29fcd64f7bbb04286404ef09de): 4586 bytes on disk
I20240502 14:28:35.308446 18901 maintenance_manager.cc:643] P c902805060df4df9b5599d0e94b2dc06: UndoDeltaBlockGCOp(8a2bed29fcd64f7bbb04286404ef09de) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4}
I20240502 14:28:35.340242 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:35.357060 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:35.362373 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:35.391348 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:35.462383 18638 maintenance_manager.cc:643] P 2db304b538e14857b7b1cc272d0db94b: FlushMRSOp(b315327141954b53a0f87c2d1dc14b9b) complete. Timing: real 0.183s user 0.055s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":10800,"dirs.run_cpu_time_us":261,"dirs.run_wall_time_us":3721,"drs_written":1,"lbm_read_time_us":178,"lbm_reads_lt_1ms":4,"lbm_write_time_us":36155,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":26912,"threads_started":2,"wal-append.queue_time_us":15015}
I20240502 14:28:35.463840 18748 maintenance_manager.cc:419] P 2db304b538e14857b7b1cc272d0db94b: Scheduling FlushMRSOp(e5fb3b049ba94af998791640bd0e31c5): perf score=0.033421
I20240502 14:28:35.528836 18638 maintenance_manager.cc:643] P 2db304b538e14857b7b1cc272d0db94b: FlushMRSOp(e5fb3b049ba94af998791640bd0e31c5) complete. Timing: real 0.065s user 0.038s sys 0.020s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":50,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":1412,"drs_written":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4,"lbm_write_time_us":895,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":10247,"threads_started":1,"wal-append.queue_time_us":10369}
I20240502 14:28:35.530118 18748 maintenance_manager.cc:419] P 2db304b538e14857b7b1cc272d0db94b: Scheduling UndoDeltaBlockGCOp(b315327141954b53a0f87c2d1dc14b9b): 4925 bytes on disk
I20240502 14:28:35.530759 18638 maintenance_manager.cc:643] P 2db304b538e14857b7b1cc272d0db94b: UndoDeltaBlockGCOp(b315327141954b53a0f87c2d1dc14b9b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240502 14:28:35.532176 18748 maintenance_manager.cc:419] P 2db304b538e14857b7b1cc272d0db94b: Scheduling UndoDeltaBlockGCOp(e5fb3b049ba94af998791640bd0e31c5): 4840 bytes on disk
I20240502 14:28:35.532727 18638 maintenance_manager.cc:643] P 2db304b538e14857b7b1cc272d0db94b: UndoDeltaBlockGCOp(e5fb3b049ba94af998791640bd0e31c5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
W20240502 14:28:35.575518 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:35.594864 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:28:35.672785 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:35.720417 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:35.742385 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:35.745424 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:35.761286 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:35.779881 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:35.792906 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:35.831602 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:28:35.872946 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:28:35.913667 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:35.916003 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:28:35.928920 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:35.962428 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:35.966733 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:35.976074 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240502 14:28:35.977919 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:35.982650 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:35.992012 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 14:28:36.115833 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:36.141932 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:28:36.146819 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240502 14:28:36.175524 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:36.176606 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:36.181515 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:36.185695 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:36.195963 18880 maintenance_manager.cc:419] P fec04c50dc5b49878ffa3c3873df32e4: Scheduling FlushMRSOp(8b8d81c275144f898dcc69a6d98d11e0): perf score=0.033343
I20240502 14:28:36.214339 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:36.231521 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:36.235303 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:36.237602 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:28:36.286876 18748 maintenance_manager.cc:419] P 2db304b538e14857b7b1cc272d0db94b: Scheduling FlushMRSOp(df5f14206a1d46e4866f34fa234f945b): perf score=0.033367
I20240502 14:28:36.296838 19143 maintenance_manager.cc:419] P 007b2e79dc424a6f80141180f072f2d8: Scheduling FlushMRSOp(df5f14206a1d46e4866f34fa234f945b): perf score=0.033372
W20240502 14:28:36.340639 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:28:36.371604 18770 maintenance_manager.cc:643] P fec04c50dc5b49878ffa3c3873df32e4: FlushMRSOp(8b8d81c275144f898dcc69a6d98d11e0) complete. Timing: real 0.175s user 0.064s sys 0.037s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":29075,"dirs.run_cpu_time_us":252,"dirs.run_wall_time_us":16701,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":812,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":54138,"threads_started":2,"wal-append.queue_time_us":25288}
I20240502 14:28:36.373728 18880 maintenance_manager.cc:419] P fec04c50dc5b49878ffa3c3873df32e4: Scheduling FlushMRSOp(5be0d6f4f1874c79b906c4bdc34b0ce9): perf score=0.033386
I20240502 14:28:36.401481 19033 maintenance_manager.cc:643] P 007b2e79dc424a6f80141180f072f2d8: FlushMRSOp(df5f14206a1d46e4866f34fa234f945b) complete. Timing: real 0.104s user 0.043s sys 0.022s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":26735,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":5640,"drs_written":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4,"lbm_write_time_us":962,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":34518,"threads_started":2,"wal-append.queue_time_us":7734}
I20240502 14:28:36.403053 19143 maintenance_manager.cc:419] P 007b2e79dc424a6f80141180f072f2d8: Scheduling FlushMRSOp(5be0d6f4f1874c79b906c4bdc34b0ce9): perf score=0.033395
W20240502 14:28:36.413223 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 14:28:36.445688 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:28:36.461844 18770 maintenance_manager.cc:643] P fec04c50dc5b49878ffa3c3873df32e4: FlushMRSOp(5be0d6f4f1874c79b906c4bdc34b0ce9) complete. Timing: real 0.088s user 0.063s sys 0.018s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":1207,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":904,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":10905,"threads_started":1,"wal-append.queue_time_us":10986}
I20240502 14:28:36.463446 18880 maintenance_manager.cc:419] P fec04c50dc5b49878ffa3c3873df32e4: Scheduling UndoDeltaBlockGCOp(8b8d81c275144f898dcc69a6d98d11e0): 4925 bytes on disk
I20240502 14:28:36.464435 18770 maintenance_manager.cc:643] P fec04c50dc5b49878ffa3c3873df32e4: UndoDeltaBlockGCOp(8b8d81c275144f898dcc69a6d98d11e0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":156,"lbm_reads_lt_1ms":4}
I20240502 14:28:36.465674 18880 maintenance_manager.cc:419] P fec04c50dc5b49878ffa3c3873df32e4: Scheduling UndoDeltaBlockGCOp(5be0d6f4f1874c79b906c4bdc34b0ce9): 4586 bytes on disk
I20240502 14:28:36.467835 18770 maintenance_manager.cc:643] P fec04c50dc5b49878ffa3c3873df32e4: UndoDeltaBlockGCOp(5be0d6f4f1874c79b906c4bdc34b0ce9) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":133,"lbm_reads_lt_1ms":4}
I20240502 14:28:36.475414 19033 maintenance_manager.cc:643] P 007b2e79dc424a6f80141180f072f2d8: FlushMRSOp(5be0d6f4f1874c79b906c4bdc34b0ce9) complete. Timing: real 0.072s user 0.038s sys 0.016s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":61,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":1463,"drs_written":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1009,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":10798,"threads_started":1,"wal-append.queue_time_us":10679}
I20240502 14:28:36.476613 19143 maintenance_manager.cc:419] P 007b2e79dc424a6f80141180f072f2d8: Scheduling UndoDeltaBlockGCOp(df5f14206a1d46e4866f34fa234f945b): 4838 bytes on disk
I20240502 14:28:36.477186 19033 maintenance_manager.cc:643] P 007b2e79dc424a6f80141180f072f2d8: UndoDeltaBlockGCOp(df5f14206a1d46e4866f34fa234f945b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20240502 14:28:36.460777 18638 maintenance_manager.cc:643] P 2db304b538e14857b7b1cc272d0db94b: FlushMRSOp(df5f14206a1d46e4866f34fa234f945b) complete. Timing: real 0.173s user 0.064s sys 0.029s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":51895,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":3799,"drs_written":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1428,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":59167,"threads_started":2,"wal-append.queue_time_us":2014}
I20240502 14:28:36.478274 19143 maintenance_manager.cc:419] P 007b2e79dc424a6f80141180f072f2d8: Scheduling UndoDeltaBlockGCOp(5be0d6f4f1874c79b906c4bdc34b0ce9): 4586 bytes on disk
I20240502 14:28:36.478686 19033 maintenance_manager.cc:643] P 007b2e79dc424a6f80141180f072f2d8: UndoDeltaBlockGCOp(5be0d6f4f1874c79b906c4bdc34b0ce9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240502 14:28:36.479830 18748 maintenance_manager.cc:419] P 2db304b538e14857b7b1cc272d0db94b: Scheduling UndoDeltaBlockGCOp(df5f14206a1d46e4866f34fa234f945b): 4838 bytes on disk
I20240502 14:28:36.485337 18638 maintenance_manager.cc:643] P 2db304b538e14857b7b1cc272d0db94b: UndoDeltaBlockGCOp(df5f14206a1d46e4866f34fa234f945b) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":165,"lbm_reads_lt_1ms":4}
W20240502 14:28:36.609011 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:28:36.616504 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:28:36.678210 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:36.690230 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:36.691191 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:36.697211 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:36.702184 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:28:36.712833 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:36.725378 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:36.754637 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:36.758179 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:28:36.777325 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:36.803961 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:36.840914 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:36.864089 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:36.872116 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:28:36.873430 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:36.890532 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:36.892863 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:36.919880 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:28:36.942319 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:28:37.056212 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:28:37.075958 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:37.081174 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:37.097198 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:37.098349 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:37.111326 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:37.115762 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:37.160560 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:28:37.161015 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:28:37.208760 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:28:37.223762 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:28:37.228701 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:28:37.235985 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:37.294462 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:37.299214 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:37.301154 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:37.305600 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:37.311297 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:37.313834 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:37.337767 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:28:37.338423 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:37.393934 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:28:37.432507 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:28:37.488250 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:28:37.490020 18748 maintenance_manager.cc:419] P 2db304b538e14857b7b1cc272d0db94b: Scheduling FlushMRSOp(db56136f2dbb429098d4d78b15a3a439): perf score=0.033340
I20240502 14:28:37.572773 19012 maintenance_manager.cc:419] P c902805060df4df9b5599d0e94b2dc06: Scheduling FlushMRSOp(51ef0be93c914e258635e13f3cca79aa): perf score=0.033363
I20240502 14:28:37.597184 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:37.639829 18901 maintenance_manager.cc:643] P c902805060df4df9b5599d0e94b2dc06: FlushMRSOp(51ef0be93c914e258635e13f3cca79aa) complete. Timing: real 0.067s user 0.035s sys 0.013s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":2274,"dirs.run_cpu_time_us":254,"dirs.run_wall_time_us":5551,"drs_written":1,"lbm_read_time_us":129,"lbm_reads_lt_1ms":4,"lbm_write_time_us":858,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":6830,"threads_started":2,"wal-append.queue_time_us":4281}
I20240502 14:28:37.641484 19012 maintenance_manager.cc:419] P c902805060df4df9b5599d0e94b2dc06: Scheduling UndoDeltaBlockGCOp(51ef0be93c914e258635e13f3cca79aa): 4925 bytes on disk
I20240502 14:28:37.642180 18901 maintenance_manager.cc:643] P c902805060df4df9b5599d0e94b2dc06: UndoDeltaBlockGCOp(51ef0be93c914e258635e13f3cca79aa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240502 14:28:37.643378 19012 maintenance_manager.cc:419] P c902805060df4df9b5599d0e94b2dc06: Scheduling FlushMRSOp(21af917743444f2db932636a78f33512): perf score=0.033378
W20240502 14:28:37.646328 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:28:37.647738 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:37.650616 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:37.668157 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:37.712549 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:37.725497 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:28:37.730126 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:37.749871 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:37.755242 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:37.755232 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:28:37.775503 18638 maintenance_manager.cc:643] P 2db304b538e14857b7b1cc272d0db94b: FlushMRSOp(db56136f2dbb429098d4d78b15a3a439) complete. Timing: real 0.285s user 0.075s sys 0.014s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":9930,"dirs.run_cpu_time_us":277,"dirs.run_wall_time_us":1553,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1010,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":32111,"threads_started":2,"wal-append.queue_time_us":3359}
I20240502 14:28:37.776984 18748 maintenance_manager.cc:419] P 2db304b538e14857b7b1cc272d0db94b: Scheduling UndoDeltaBlockGCOp(db56136f2dbb429098d4d78b15a3a439): 4586 bytes on disk
I20240502 14:28:37.777738 18638 maintenance_manager.cc:643] P 2db304b538e14857b7b1cc272d0db94b: UndoDeltaBlockGCOp(db56136f2dbb429098d4d78b15a3a439) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":138,"lbm_reads_lt_1ms":4}
I20240502 14:28:37.788653 18901 maintenance_manager.cc:643] P c902805060df4df9b5599d0e94b2dc06: FlushMRSOp(21af917743444f2db932636a78f33512) complete. Timing: real 0.145s user 0.044s sys 0.011s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":14940,"dirs.run_cpu_time_us":238,"dirs.run_wall_time_us":1888,"drs_written":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1052,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":6416,"threads_started":1,"wal-append.queue_time_us":6531}
I20240502 14:28:37.790887 19012 maintenance_manager.cc:419] P c902805060df4df9b5599d0e94b2dc06: Scheduling UndoDeltaBlockGCOp(21af917743444f2db932636a78f33512): 4840 bytes on disk
I20240502 14:28:37.791864 18901 maintenance_manager.cc:643] P c902805060df4df9b5599d0e94b2dc06: UndoDeltaBlockGCOp(21af917743444f2db932636a78f33512) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":183,"lbm_reads_lt_1ms":4}
W20240502 14:28:37.846469 20562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:28:37.847851 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:28:37.918598 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:37.921763 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:37.930490 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:37.957281 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:37.988754 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:38.015663 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:38.165791 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:38.180584 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:38.181813 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:38.188822 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:28:38.205796 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:38.217419 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:28:38.226635 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:38.229102 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:38.239828 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:38.307895 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:38.400540 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:28:38.442829 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:28:38.475281 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:38.476174 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:38.482445 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:38.485240 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:38.485461 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:38.491533 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:38.611977 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:28:38.612100 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:38.672400 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:38.750578 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:38.756315 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:38.765846 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:38.775132 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:38.782446 18748 maintenance_manager.cc:419] P 2db304b538e14857b7b1cc272d0db94b: Scheduling FlushMRSOp(1ecc6539b85d40e49df7ffc5f93cb801): perf score=0.033340
I20240502 14:28:38.782894 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:38.787107 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:38.883684 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:28:38.891690 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:28:38.895287 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:28:38.927376 18638 maintenance_manager.cc:643] P 2db304b538e14857b7b1cc272d0db94b: FlushMRSOp(1ecc6539b85d40e49df7ffc5f93cb801) complete. Timing: real 0.145s user 0.048s sys 0.020s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":313,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":1584,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1033,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":504,"threads_started":2,"wal-append.queue_time_us":362}
I20240502 14:28:38.928640 18748 maintenance_manager.cc:419] P 2db304b538e14857b7b1cc272d0db94b: Scheduling UndoDeltaBlockGCOp(1ecc6539b85d40e49df7ffc5f93cb801): 4925 bytes on disk
I20240502 14:28:38.929265 18638 maintenance_manager.cc:643] P 2db304b538e14857b7b1cc272d0db94b: UndoDeltaBlockGCOp(1ecc6539b85d40e49df7ffc5f93cb801) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240502 14:28:38.979952 18880 maintenance_manager.cc:419] P fec04c50dc5b49878ffa3c3873df32e4: Scheduling FlushMRSOp(2c2afaffde96462eacd0d32bbdec23b1): perf score=0.033383
I20240502 14:28:38.988916 19143 maintenance_manager.cc:419] P 007b2e79dc424a6f80141180f072f2d8: Scheduling FlushMRSOp(246a8425400f4890a5264c4ae8a627c5): perf score=0.033387
W20240502 14:28:39.015231 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:28:39.032869 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:39.049474 19012 maintenance_manager.cc:419] P c902805060df4df9b5599d0e94b2dc06: Scheduling FlushMRSOp(246a8425400f4890a5264c4ae8a627c5): perf score=0.033398
I20240502 14:28:39.064484 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:39.109169 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:39.126271 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:39.128916 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:39.131171 18770 maintenance_manager.cc:643] P fec04c50dc5b49878ffa3c3873df32e4: FlushMRSOp(2c2afaffde96462eacd0d32bbdec23b1) complete. Timing: real 0.151s user 0.041s sys 0.017s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":4980,"dirs.run_cpu_time_us":219,"dirs.run_wall_time_us":6548,"drs_written":1,"lbm_read_time_us":123,"lbm_reads_lt_1ms":4,"lbm_write_time_us":927,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":33723,"threads_started":2,"wal-append.queue_time_us":28691}
I20240502 14:28:39.132580 18880 maintenance_manager.cc:419] P fec04c50dc5b49878ffa3c3873df32e4: Scheduling UndoDeltaBlockGCOp(2c2afaffde96462eacd0d32bbdec23b1): 4840 bytes on disk
I20240502 14:28:39.133219 18770 maintenance_manager.cc:643] P fec04c50dc5b49878ffa3c3873df32e4: UndoDeltaBlockGCOp(2c2afaffde96462eacd0d32bbdec23b1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240502 14:28:39.136641 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:39.150272 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:39.162576 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:39.213538 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:28:39.214694 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:39.222033 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:28:39.224788 18901 maintenance_manager.cc:643] P c902805060df4df9b5599d0e94b2dc06: FlushMRSOp(246a8425400f4890a5264c4ae8a627c5) complete. Timing: real 0.175s user 0.043s sys 0.009s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":11074,"dirs.run_cpu_time_us":254,"dirs.run_wall_time_us":11482,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":13581,"lbm_writes_1-10_ms":4,"lbm_writes_lt_1ms":19,"peak_mem_usage":0,"rows_written":635,"thread_start_us":52307,"threads_started":2,"wal-append.queue_time_us":34029}
I20240502 14:28:39.231379 19012 maintenance_manager.cc:419] P c902805060df4df9b5599d0e94b2dc06: Scheduling UndoDeltaBlockGCOp(246a8425400f4890a5264c4ae8a627c5): 4586 bytes on disk
I20240502 14:28:39.234975 18901 maintenance_manager.cc:643] P c902805060df4df9b5599d0e94b2dc06: UndoDeltaBlockGCOp(246a8425400f4890a5264c4ae8a627c5) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":144,"lbm_reads_lt_1ms":4}
I20240502 14:28:39.238171 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:39.266947 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:28:39.311925 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:39.319609 19033 maintenance_manager.cc:643] P 007b2e79dc424a6f80141180f072f2d8: FlushMRSOp(246a8425400f4890a5264c4ae8a627c5) complete. Timing: real 0.330s user 0.073s sys 0.055s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":41464,"dirs.run_cpu_time_us":259,"dirs.run_wall_time_us":7141,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":64618,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":151026,"threads_started":2,"wal-append.queue_time_us":108812}
I20240502 14:28:39.320932 19143 maintenance_manager.cc:419] P 007b2e79dc424a6f80141180f072f2d8: Scheduling UndoDeltaBlockGCOp(246a8425400f4890a5264c4ae8a627c5): 4586 bytes on disk
I20240502 14:28:39.326056 19033 maintenance_manager.cc:643] P 007b2e79dc424a6f80141180f072f2d8: UndoDeltaBlockGCOp(246a8425400f4890a5264c4ae8a627c5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
W20240502 14:28:39.328899 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:28:39.341683 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:39.342100 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:39.342825 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:28:39.347553 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:39.450410 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:28:39.520541 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:39.523917 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:39.547973 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:39.569928 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:39.573874 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:39.577821 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:39.598886 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:28:39.603669 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:39.608850 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:39.627748 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:28:39.638741 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:28:39.747447 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:28:39.764379 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:28:39.800930 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:28:39.824090 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:39.848855 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:39.853927 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:39.858528 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:39.860077 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:39.892760 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:39.893370 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:28:39.914662 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:28:39.983204 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:39.988638 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:40.000057 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:28:40.041148 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:28:40.042567 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:40.042574 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:40.054906 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:40.063791 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:40.081668 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:40.111799 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:40.173980 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:28:40.181905 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:28:40.241138 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:40.258702 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:40.279417 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:40.285634 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:40.293742 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:40.295329 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:40.379758 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:28:40.413462 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:28:40.528152 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:28:40.531930 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:40.540112 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:40.553256 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:40.563284 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:40.565817 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:40.570353 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:40.593079 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:40.652848 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:40.699203 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:28:40.848790 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:40.862095 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:40.872969 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:40.885608 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:40.902153 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:40.932626 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:40.971521 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:40.998463 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240502 14:28:41.044404 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:41.052141 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:41.071280 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:41.079313 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:41.097112 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:41.098233 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 14:28:41.114725 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:28:41.115957 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:41.238257 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:28:41.272116 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:41.293658 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:41.321461 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:41.324060 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:41.327126 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:41.328778 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:41.340920 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240502 14:28:41.354043 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:41.365525 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:41.413206 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:28:41.497422 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:41.503607 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:41.513500 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:41.519150 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:41.523007 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:41.547668 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:41.560163 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 14:28:41.563694 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:28:41.588608 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:28:41.666801 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:28:41.730587 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:28:41.780222 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:41.796792 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:41.804520 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:41.825623 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:41.829943 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:41.834583 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:28:41.842136 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:41.857103 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:41.858106 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:41.889796 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:28:41.908497 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:28:41.939595 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:28:41.985850 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:28:42.024093 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:28:42.026194 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:28:42.071936 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:42.093956 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:42.106654 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:42.114475 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:42.138070 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:42.154486 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:42.156286 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:28:42.209076 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:28:42.251125 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:28:42.279024 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:42.305121 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:42.324743 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:42.331460 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:28:42.335160 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:42.342085 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:42.343240 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:42.350772 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:42.383534 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:28:42.420110 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:28:42.420655 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:42.436625 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:28:42.531909 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:42.545749 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:42.556084 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:28:42.579416 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:42.590926 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:42.609269 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:28:42.615797 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:28:42.623945 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:42.637610 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:42.812959 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:42.813578 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:28:42.849858 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:28:42.851178 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:42.861946 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:42.882920 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:42.891094 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:28:42.928333 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:42.994947 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:43.006019 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:43.032665 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:28:43.034952 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:43.062268 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:28:43.077795 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:43.109437 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:43.132505 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:43.140266 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:43.158032 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 14:28:43.170441 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:43.175812 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:43.289932 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:43.337749 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:43.343366 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:43.354265 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:43.355664 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:43.357425 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:43.369663 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:43.414923 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:43.558743 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:28:43.595567 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:43.605561 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:43.627969 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:43.628904 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:43.703315 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:43.704890 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:43.705009 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:28:43.802098 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:28:43.821593 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:43.901422 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:43.905165 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:43.913108 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:43.921061 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:43.933070 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:43.960142 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:43.988054 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:28:43.997987 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:44.099601 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:28:44.121317 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:44.130501 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:44.157667 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:44.170787 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:44.200314 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:44.216024 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:44.236343 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:28:44.238433 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:44.338095 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:28:44.376641 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:44.394569 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:44.415681 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:28:44.463459 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:44.476043 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:44.509406 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:28:44.520956 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:44.522008 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:44.532204 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:44.578016 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:28:44.627422 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:28:44.669751 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:28:44.676225 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:44.690456 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:28:44.692150 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:44.715554 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:44.722748 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:28:44.728008 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:44.728798 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:44.750015 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:28:44.763373 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:28:44.780099 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:44.802223 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:44.852840 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:28:44.878468 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:44.902565 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:28:44.925124 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:28:44.962838 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:44.969592 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:44.970502 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:44.975762 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:44.985877 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:44.994728 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:45.056406 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:28:45.061762 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:28:45.070113 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:28:45.146867 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:28:45.195119 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:28:45.227006 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:28:45.262245 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:45.264096 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:45.299463 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:45.310487 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:28:45.311405 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:45.321132 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:45.356103 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:45.359985 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:45.425153 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:45.446475 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:45.449648 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:45.476527 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:45.505460 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:45.536882 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:45.583555 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:28:45.587392 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:45.604614 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:28:45.638875 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:45.783946 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:45.795639 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:45.799988 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:45.808677 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:45.822361 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:45.824841 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:45.827037 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:46.050904 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:28:46.069509 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:46.082664 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240502 14:28:46.085536 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:46.088775 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:46.118511 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:46.128436 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:46.132697 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:46.158457 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 14:28:46.179273 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:28:46.256578 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:46.300844 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:46.306140 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:46.307813 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:46.321543 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:46.327044 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:46.337558 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:46.355894 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 14:28:46.378671 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:46.460839 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:28:46.502727 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:46.516249 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:46.531536 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:46.544273 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:46.551589 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:46.576997 18768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.72:35871, user_credentials={real_user=slave}} blocked reactor thread for 50507.8us
I20240502 14:28:46.581887 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:46.653023 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:46.671002 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 14:28:46.671944 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:28:46.676759 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:46.704607 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:28:46.729089 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:28:46.779469 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:28:46.780290 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:46.790232 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:46.798719 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:46.804631 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:46.806061 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:46.811105 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:46.917944 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:28:46.973982 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:28:47.034315 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:28:47.052071 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:47.072083 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:28:47.082845 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:28:47.083835 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:47.112720 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:47.122465 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:47.147650 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:28:47.161382 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:47.161983 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:47.186100 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:47.196265 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:47.238045 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:28:47.247567 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:47.270403 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:28:47.273790 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:47.273885 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:28:47.294965 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:28:47.297667 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:47.305790 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:47.310451 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:28:47.321874 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:47.325120 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:47.356868 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:28:47.565308 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:47.591197 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:47.597016 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:28:47.614920 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:28:47.627084 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:47.642563 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:47.650410 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:28:47.657948 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:47.658641 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:47.701256 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:28:47.709235 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:28:47.711706 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:47.722637 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:47.736948 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:28:47.756743 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:47.772539 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:28:47.778909 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:47.832762 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:47.857539 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:47.858459 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:47.866797 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:48.263446 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:28:48.278262 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:28:48.288585 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:28:48.288853 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:28:48.296392 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:48.344101 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:48.405046 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:48.411923 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:48.435945 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:48.440742 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:48.470360 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:28:48.472470 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:48.510824 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:48.523636 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:48.546337 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:48.565961 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:48.579155 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:48.588438 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:48.617915 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:48.738461 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:48.738539 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:28:48.823976 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:28:48.824136 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:48.885103 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:48.906659 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:48.928336 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:28:48.930449 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:48.946707 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:48.969214 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:48.974910 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:48.995819 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:48.995738 18769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.68:41463, user_credentials={real_user=slave}} blocked reactor thread for 66423us
I20240502 14:28:49.022794 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:49.086874 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:49.113358 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:49.117991 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:49.133102 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:49.224397 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:28:49.261329 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:28:49.307022 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:49.346185 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:49.367959 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:49.370509 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:49.383226 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:49.391774 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:49.397980 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:49.424700 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:28:49.430444 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:49.473901 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:28:49.495950 20673 maintenance_manager.cc:419] P a834edffbcb2469aae598b069aab0681: Scheduling FlushMRSOp(9580f3fdf7084cc0923b1b16bfedd450): perf score=0.033389
W20240502 14:28:49.500903 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:28:49.523298 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:28:49.555546 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:28:49.573232 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:28:49.607818 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:28:49.616456 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:28:49.637687 20563 maintenance_manager.cc:643] P a834edffbcb2469aae598b069aab0681: FlushMRSOp(9580f3fdf7084cc0923b1b16bfedd450) complete. Timing: real 0.141s user 0.073s sys 0.025s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":347,"dirs.queue_time_us":6374,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":5051,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1166,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":21155,"threads_started":3,"wal-append.queue_time_us":9577}
I20240502 14:28:49.639758 20673 maintenance_manager.cc:419] P a834edffbcb2469aae598b069aab0681: Scheduling UndoDeltaBlockGCOp(9580f3fdf7084cc0923b1b16bfedd450): 4586 bytes on disk
I20240502 14:28:49.640801 20563 maintenance_manager.cc:643] P a834edffbcb2469aae598b069aab0681: UndoDeltaBlockGCOp(9580f3fdf7084cc0923b1b16bfedd450) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20240502 14:28:49.677453 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:49.683765 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:49.708796 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:49.723007 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:49.735286 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:28:49.775720 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:49.789438 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:28:49.790196 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:49.845723 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:49.883497 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:49.900691 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:28:49.909996 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:28:49.967723 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:28:50.019021 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:28:50.038621 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:28:50.121943 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:28:50.173401 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:50.198832 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:50.213461 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:28:50.288612 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:28:50.291054 21077 maintenance_manager.cc:419] P ff6acba046404ef0a6f41ad5e09d852f: Scheduling FlushMRSOp(9580f3fdf7084cc0923b1b16bfedd450): perf score=0.033361
I20240502 14:28:50.321930 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:50.322693 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:50.363865 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:28:50.366961 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:50.369046 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:50.405930 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:50.421651 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:50.439236 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:28:50.539841 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:50.549608 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:50.564762 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:50.573773 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:50.582082 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:50.619906 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:28:50.622603 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:50.762917 20965 maintenance_manager.cc:643] P ff6acba046404ef0a6f41ad5e09d852f: FlushMRSOp(9580f3fdf7084cc0923b1b16bfedd450) complete. Timing: real 0.471s user 0.088s sys 0.032s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":41838,"dirs.queue_time_us":13706,"dirs.run_cpu_time_us":298,"dirs.run_wall_time_us":1165,"drs_written":1,"lbm_read_time_us":182,"lbm_reads_lt_1ms":4,"lbm_write_time_us":946,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":63206,"threads_started":3,"wal-append.queue_time_us":7342}
I20240502 14:28:50.764808 21077 maintenance_manager.cc:419] P ff6acba046404ef0a6f41ad5e09d852f: Scheduling UndoDeltaBlockGCOp(9580f3fdf7084cc0923b1b16bfedd450): 4586 bytes on disk
I20240502 14:28:50.765650 20965 maintenance_manager.cc:643] P ff6acba046404ef0a6f41ad5e09d852f: UndoDeltaBlockGCOp(9580f3fdf7084cc0923b1b16bfedd450) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240502 14:28:50.766575 21077 maintenance_manager.cc:419] P ff6acba046404ef0a6f41ad5e09d852f: Scheduling FlushMRSOp(332e4b23751b422ab7bd17b80c93701d): perf score=0.033479
I20240502 14:28:50.807703 20965 maintenance_manager.cc:643] P ff6acba046404ef0a6f41ad5e09d852f: FlushMRSOp(332e4b23751b422ab7bd17b80c93701d) complete. Timing: real 0.041s user 0.028s sys 0.012s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":68,"dirs.run_cpu_time_us":246,"dirs.run_wall_time_us":1482,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":671,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6284,"threads_started":1,"wal-append.queue_time_us":6407}
I20240502 14:28:50.809422 21077 maintenance_manager.cc:419] P ff6acba046404ef0a6f41ad5e09d852f: Scheduling UndoDeltaBlockGCOp(332e4b23751b422ab7bd17b80c93701d): 4586 bytes on disk
I20240502 14:28:50.810073 20965 maintenance_manager.cc:643] P ff6acba046404ef0a6f41ad5e09d852f: UndoDeltaBlockGCOp(332e4b23751b422ab7bd17b80c93701d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
W20240502 14:28:50.820071 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:28:50.914219 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:50.915009 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:50.942440 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:50.943477 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:50.947836 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:50.949661 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:50.977695 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:50.978020 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:51.050386 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:28:51.062222 21077 maintenance_manager.cc:419] P ff6acba046404ef0a6f41ad5e09d852f: Scheduling FlushMRSOp(8937c809610542d88fdde3daf002c890): perf score=0.033379
W20240502 14:28:51.121803 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 14:28:51.158861 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 14:28:51.160300 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:28:51.193094 20965 maintenance_manager.cc:643] P ff6acba046404ef0a6f41ad5e09d852f: FlushMRSOp(8937c809610542d88fdde3daf002c890) complete. Timing: real 0.130s user 0.057s sys 0.019s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":216,"dirs.run_wall_time_us":1619,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":978,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":14000,"threads_started":1,"wal-append.queue_time_us":12688}
I20240502 14:28:51.194519 21077 maintenance_manager.cc:419] P ff6acba046404ef0a6f41ad5e09d852f: Scheduling UndoDeltaBlockGCOp(8937c809610542d88fdde3daf002c890): 4925 bytes on disk
I20240502 14:28:51.195250 20965 maintenance_manager.cc:643] P ff6acba046404ef0a6f41ad5e09d852f: UndoDeltaBlockGCOp(8937c809610542d88fdde3daf002c890) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240502 14:28:51.196352 21077 maintenance_manager.cc:419] P ff6acba046404ef0a6f41ad5e09d852f: Scheduling FlushMRSOp(0ffb6e582ef04fb38dd273c13febac58): perf score=0.033336
I20240502 14:28:51.296770 20965 maintenance_manager.cc:643] P ff6acba046404ef0a6f41ad5e09d852f: FlushMRSOp(0ffb6e582ef04fb38dd273c13febac58) complete. Timing: real 0.100s user 0.059s sys 0.017s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":50,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":1531,"drs_written":1,"lbm_read_time_us":122,"lbm_reads_lt_1ms":4,"lbm_write_time_us":811,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":4550,"threads_started":1,"wal-append.queue_time_us":4452}
I20240502 14:28:51.299010 21077 maintenance_manager.cc:419] P ff6acba046404ef0a6f41ad5e09d852f: Scheduling UndoDeltaBlockGCOp(0ffb6e582ef04fb38dd273c13febac58): 4840 bytes on disk
I20240502 14:28:51.301704 20965 maintenance_manager.cc:643] P ff6acba046404ef0a6f41ad5e09d852f: UndoDeltaBlockGCOp(0ffb6e582ef04fb38dd273c13febac58) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4}
W20240502 14:28:51.411960 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:51.468977 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:51.499552 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:51.515473 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:28:51.533433 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:51.558916 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:28:51.567366 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 14:28:51.569468 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:51.600147 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:51.600417 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:51.616870 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:51.652210 20673 maintenance_manager.cc:419] P a834edffbcb2469aae598b069aab0681: Scheduling FlushMRSOp(8937c809610542d88fdde3daf002c890): perf score=0.033375
I20240502 14:28:51.653575 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:51.659359 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:28:51.667165 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:51.682875 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:51.684054 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:51.691143 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:51.694679 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:51.804533 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 14:28:51.806483 21077 maintenance_manager.cc:419] P ff6acba046404ef0a6f41ad5e09d852f: Scheduling FlushMRSOp(522014370f0a4cc3bd31aa00d6a05c5e): perf score=0.033348
I20240502 14:28:51.835381 20563 maintenance_manager.cc:643] P a834edffbcb2469aae598b069aab0681: FlushMRSOp(8937c809610542d88fdde3daf002c890) complete. Timing: real 0.183s user 0.071s sys 0.033s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":21804,"dirs.run_cpu_time_us":276,"dirs.run_wall_time_us":1616,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":745,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":49511,"threads_started":2,"wal-append.queue_time_us":18704}
I20240502 14:28:51.837551 20673 maintenance_manager.cc:419] P a834edffbcb2469aae598b069aab0681: Scheduling UndoDeltaBlockGCOp(8937c809610542d88fdde3daf002c890): 4925 bytes on disk
W20240502 14:28:51.838330 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:28:51.854251 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:28:51.854815 20563 maintenance_manager.cc:643] P a834edffbcb2469aae598b069aab0681: UndoDeltaBlockGCOp(8937c809610542d88fdde3daf002c890) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
I20240502 14:28:51.893594 20965 maintenance_manager.cc:643] P ff6acba046404ef0a6f41ad5e09d852f: FlushMRSOp(522014370f0a4cc3bd31aa00d6a05c5e) complete. Timing: real 0.087s user 0.046s sys 0.002s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":949,"dirs.run_cpu_time_us":270,"dirs.run_wall_time_us":1561,"drs_written":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4,"lbm_write_time_us":849,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":6681,"threads_started":2,"wal-append.queue_time_us":366}
I20240502 14:28:51.895202 21077 maintenance_manager.cc:419] P ff6acba046404ef0a6f41ad5e09d852f: Scheduling UndoDeltaBlockGCOp(522014370f0a4cc3bd31aa00d6a05c5e): 4840 bytes on disk
I20240502 14:28:51.895923 20965 maintenance_manager.cc:643] P ff6acba046404ef0a6f41ad5e09d852f: UndoDeltaBlockGCOp(522014370f0a4cc3bd31aa00d6a05c5e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":148,"lbm_reads_lt_1ms":4}
W20240502 14:28:51.951949 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:28:51.964293 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:28:51.980752 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:28:51.997869 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:52.006752 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:52.015146 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:52.017303 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:52.020457 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:52.021075 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:52.041054 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:28:52.048058 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:52.050522 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:28:52.089310 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:52.136222 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:28:52.148099 21077 maintenance_manager.cc:419] P ff6acba046404ef0a6f41ad5e09d852f: Scheduling FlushMRSOp(b315327141954b53a0f87c2d1dc14b9b): perf score=0.033364
I20240502 14:28:52.182957 20965 maintenance_manager.cc:643] P ff6acba046404ef0a6f41ad5e09d852f: FlushMRSOp(b315327141954b53a0f87c2d1dc14b9b) complete. Timing: real 0.035s user 0.029s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":65,"dirs.run_cpu_time_us":238,"dirs.run_wall_time_us":1651,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":642,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":250,"threads_started":1,"wal-append.queue_time_us":462}
I20240502 14:28:52.184433 21077 maintenance_manager.cc:419] P ff6acba046404ef0a6f41ad5e09d852f: Scheduling UndoDeltaBlockGCOp(b315327141954b53a0f87c2d1dc14b9b): 4925 bytes on disk
I20240502 14:28:52.184983 20965 maintenance_manager.cc:643] P ff6acba046404ef0a6f41ad5e09d852f: UndoDeltaBlockGCOp(b315327141954b53a0f87c2d1dc14b9b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
W20240502 14:28:52.273797 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:28:52.283177 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:52.287853 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:52.288470 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:52.288784 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:28:52.288875 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:52.292699 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:52.318169 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:52.344009 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:28:52.345381 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:28:52.411602 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:28:52.416115 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:52.419150 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:28:52.465135 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:52.510308 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:28:52.540802 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:28:52.624534 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:52.654527 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:52.667572 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:52.680908 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:52.694794 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:52.705016 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:52.723449 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:52.725593 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:52.748687 18636 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.72:35871, user_credentials={real_user=slave}} blocked reactor thread for 62228us
W20240502 14:28:52.762653 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:28:52.764605 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:28:52.766752 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:52.784904 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:52.793964 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:52.807174 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:52.813663 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:28:52.861901 20673 maintenance_manager.cc:419] P a834edffbcb2469aae598b069aab0681: Scheduling FlushMRSOp(8a2bed29fcd64f7bbb04286404ef09de): perf score=0.033388
W20240502 14:28:52.880615 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:28:52.918087 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:52.927731 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:52.939671 21077 maintenance_manager.cc:419] P ff6acba046404ef0a6f41ad5e09d852f: Scheduling FlushMRSOp(0d8667d91fd64809b635362c4a8fa33f): perf score=0.033351
W20240502 14:28:52.972723 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:28:52.981216 20563 maintenance_manager.cc:643] P a834edffbcb2469aae598b069aab0681: FlushMRSOp(8a2bed29fcd64f7bbb04286404ef09de) complete. Timing: real 0.119s user 0.061s sys 0.020s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6927,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":6335,"drs_written":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4,"lbm_write_time_us":753,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":13771,"threads_started":2,"wal-append.queue_time_us":6999}
I20240502 14:28:52.983435 20673 maintenance_manager.cc:419] P a834edffbcb2469aae598b069aab0681: Scheduling FlushMRSOp(df5f14206a1d46e4866f34fa234f945b): perf score=0.033409
I20240502 14:28:53.015985 20965 maintenance_manager.cc:643] P ff6acba046404ef0a6f41ad5e09d852f: FlushMRSOp(0d8667d91fd64809b635362c4a8fa33f) complete. Timing: real 0.076s user 0.045s sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6385,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":9024,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":799,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":13886,"threads_started":2,"wal-append.queue_time_us":7294}
I20240502 14:28:53.017666 21077 maintenance_manager.cc:419] P ff6acba046404ef0a6f41ad5e09d852f: Scheduling UndoDeltaBlockGCOp(0d8667d91fd64809b635362c4a8fa33f): 4925 bytes on disk
I20240502 14:28:53.018678 20965 maintenance_manager.cc:643] P ff6acba046404ef0a6f41ad5e09d852f: UndoDeltaBlockGCOp(0d8667d91fd64809b635362c4a8fa33f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
I20240502 14:28:53.044389 20563 maintenance_manager.cc:643] P a834edffbcb2469aae598b069aab0681: FlushMRSOp(df5f14206a1d46e4866f34fa234f945b) complete. Timing: real 0.061s user 0.051s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":1864,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":2683,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1080,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":1749,"threads_started":1,"wal-append.queue_time_us":1654}
I20240502 14:28:53.046187 20673 maintenance_manager.cc:419] P a834edffbcb2469aae598b069aab0681: Scheduling UndoDeltaBlockGCOp(df5f14206a1d46e4866f34fa234f945b): 4838 bytes on disk
I20240502 14:28:53.046911 20563 maintenance_manager.cc:643] P a834edffbcb2469aae598b069aab0681: UndoDeltaBlockGCOp(df5f14206a1d46e4866f34fa234f945b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":124,"lbm_reads_lt_1ms":4}
I20240502 14:28:53.047956 20673 maintenance_manager.cc:419] P a834edffbcb2469aae598b069aab0681: Scheduling UndoDeltaBlockGCOp(8a2bed29fcd64f7bbb04286404ef09de): 4586 bytes on disk
I20240502 14:28:53.048909 20563 maintenance_manager.cc:643] P a834edffbcb2469aae598b069aab0681: UndoDeltaBlockGCOp(8a2bed29fcd64f7bbb04286404ef09de) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240502 14:28:53.104769 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:53.129000 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:53.151717 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:53.162760 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:53.171191 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:28:53.192015 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:53.220820 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:53.318914 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:53.348212 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 14:28:53.353300 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:53.365214 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:53.365657 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:53.369750 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:53.381814 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:53.392566 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:53.393141 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:28:53.524662 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:28:53.534965 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:28:53.536074 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:53.643093 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:53.652925 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:53.655078 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:53.656263 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:53.673698 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:53.689571 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:53.844869 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:28:53.921696 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:53.996505 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:53.998288 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:54.023339 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:54.028489 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:28:54.034828 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:54.035528 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:54.055289 20673 maintenance_manager.cc:419] P a834edffbcb2469aae598b069aab0681: Scheduling FlushMRSOp(522014370f0a4cc3bd31aa00d6a05c5e): perf score=0.033373
I20240502 14:28:54.071058 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:54.105070 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:28:54.107386 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:54.109817 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:54.118394 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:28:54.146796 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:54.154603 20964 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.5.107.71:39195, user_credentials={real_user=slave}} blocked reactor thread for 58244.6us
W20240502 14:28:54.160637 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:54.162444 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:54.179766 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:54.212599 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:54.255848 20563 maintenance_manager.cc:643] P a834edffbcb2469aae598b069aab0681: FlushMRSOp(522014370f0a4cc3bd31aa00d6a05c5e) complete. Timing: real 0.200s user 0.038s sys 0.014s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":17491,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":31477,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":771,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":44385,"threads_started":2,"wal-append.queue_time_us":23287}
I20240502 14:28:54.262614 20673 maintenance_manager.cc:419] P a834edffbcb2469aae598b069aab0681: Scheduling UndoDeltaBlockGCOp(522014370f0a4cc3bd31aa00d6a05c5e): 4840 bytes on disk
I20240502 14:28:54.263556 20563 maintenance_manager.cc:643] P a834edffbcb2469aae598b069aab0681: UndoDeltaBlockGCOp(522014370f0a4cc3bd31aa00d6a05c5e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4}
I20240502 14:28:54.275805 21077 maintenance_manager.cc:419] P ff6acba046404ef0a6f41ad5e09d852f: Scheduling FlushMRSOp(8b8d81c275144f898dcc69a6d98d11e0): perf score=0.033401
W20240502 14:28:54.299584 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:54.311901 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:28:54.331208 20965 maintenance_manager.cc:643] P ff6acba046404ef0a6f41ad5e09d852f: FlushMRSOp(8b8d81c275144f898dcc69a6d98d11e0) complete. Timing: real 0.055s user 0.049s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":410,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":3159,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":871,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":5908,"threads_started":2,"wal-append.queue_time_us":5813}
I20240502 14:28:54.332460 21077 maintenance_manager.cc:419] P ff6acba046404ef0a6f41ad5e09d852f: Scheduling UndoDeltaBlockGCOp(8b8d81c275144f898dcc69a6d98d11e0): 4925 bytes on disk
I20240502 14:28:54.333246 20965 maintenance_manager.cc:643] P ff6acba046404ef0a6f41ad5e09d852f: UndoDeltaBlockGCOp(8b8d81c275144f898dcc69a6d98d11e0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":221,"lbm_reads_lt_1ms":4}
W20240502 14:28:54.335675 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:28:54.371122 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:28:54.371117 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:28:54.461915 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:54.468308 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:54.471485 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:54.475973 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:54.483237 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:54.485471 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:54.536789 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:54.538946 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:28:54.560379 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:28:54.599936 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:28:54.637816 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:28:54.718076 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:28:54.747740 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:28:54.765148 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:28:54.776712 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:54.788565 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:54.830374 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:54.831127 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:54.834467 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:54.835834 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:54.838526 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:54.846750 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:54.904750 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:28:54.905516 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:28:54.910164 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:28:54.914180 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:54.929801 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:54.931756 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:54.956575 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:54.962829 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:55.002797 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:55.018270 20673 maintenance_manager.cc:419] P a834edffbcb2469aae598b069aab0681: Scheduling FlushMRSOp(2c2afaffde96462eacd0d32bbdec23b1): perf score=0.033335
W20240502 14:28:55.046133 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:28:55.062950 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:28:55.088320 20563 maintenance_manager.cc:643] P a834edffbcb2469aae598b069aab0681: FlushMRSOp(2c2afaffde96462eacd0d32bbdec23b1) complete. Timing: real 0.070s user 0.035s sys 0.006s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":328,"dirs.run_cpu_time_us":334,"dirs.run_wall_time_us":1496,"drs_written":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4,"lbm_write_time_us":812,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":446,"threads_started":2,"wal-append.queue_time_us":357}
I20240502 14:28:55.089998 20673 maintenance_manager.cc:419] P a834edffbcb2469aae598b069aab0681: Scheduling FlushMRSOp(097eb705c9e24740a5be61bf3354f3d8): perf score=0.033341
W20240502 14:28:55.104703 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:28:55.148897 20563 maintenance_manager.cc:643] P a834edffbcb2469aae598b069aab0681: FlushMRSOp(097eb705c9e24740a5be61bf3354f3d8) complete. Timing: real 0.059s user 0.046s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":78,"dirs.run_cpu_time_us":279,"dirs.run_wall_time_us":1541,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":830,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":231,"threads_started":1,"wal-append.queue_time_us":342}
I20240502 14:28:55.150403 20673 maintenance_manager.cc:419] P a834edffbcb2469aae598b069aab0681: Scheduling UndoDeltaBlockGCOp(2c2afaffde96462eacd0d32bbdec23b1): 4840 bytes on disk
I20240502 14:28:55.150995 20563 maintenance_manager.cc:643] P a834edffbcb2469aae598b069aab0681: UndoDeltaBlockGCOp(2c2afaffde96462eacd0d32bbdec23b1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240502 14:28:55.151983 20673 maintenance_manager.cc:419] P a834edffbcb2469aae598b069aab0681: Scheduling UndoDeltaBlockGCOp(097eb705c9e24740a5be61bf3354f3d8): 4586 bytes on disk
I20240502 14:28:55.152606 20563 maintenance_manager.cc:643] P a834edffbcb2469aae598b069aab0681: UndoDeltaBlockGCOp(097eb705c9e24740a5be61bf3354f3d8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240502 14:28:55.232584 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:55.249554 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:28:55.249862 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:55.255597 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:55.264403 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:55.273768 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:55.276494 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:55.346354 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:55.371222 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:55.420634 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:28:55.480969 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:55.487191 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:55.508589 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:55.508760 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:55.512362 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:55.528370 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:55.552575 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:28:55.586385 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:28:55.590502 21077 maintenance_manager.cc:419] P ff6acba046404ef0a6f41ad5e09d852f: Scheduling FlushMRSOp(b15d11209e45473e863766e5cd6e822f): perf score=0.033358
I20240502 14:28:55.650833 20965 maintenance_manager.cc:643] P ff6acba046404ef0a6f41ad5e09d852f: FlushMRSOp(b15d11209e45473e863766e5cd6e822f) complete. Timing: real 0.060s user 0.041s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":514,"dirs.run_cpu_time_us":278,"dirs.run_wall_time_us":1498,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1032,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":778,"threads_started":2,"wal-append.queue_time_us":967}
I20240502 14:28:55.652496 21077 maintenance_manager.cc:419] P ff6acba046404ef0a6f41ad5e09d852f: Scheduling UndoDeltaBlockGCOp(b15d11209e45473e863766e5cd6e822f): 4925 bytes on disk
I20240502 14:28:55.653642 20965 maintenance_manager.cc:643] P ff6acba046404ef0a6f41ad5e09d852f: UndoDeltaBlockGCOp(b15d11209e45473e863766e5cd6e822f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
W20240502 14:28:55.671810 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:28:55.803675 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:55.818852 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:55.827162 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:55.849002 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:55.866673 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:55.900086 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:55.906841 20673 maintenance_manager.cc:419] P a834edffbcb2469aae598b069aab0681: Scheduling FlushMRSOp(b13bcd6e693f4c7687a1aabdafbb83c6): perf score=0.033360
W20240502 14:28:55.936316 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:28:55.961723 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:55.991616 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:56.010233 20563 maintenance_manager.cc:643] P a834edffbcb2469aae598b069aab0681: FlushMRSOp(b13bcd6e693f4c7687a1aabdafbb83c6) complete. Timing: real 0.103s user 0.041s sys 0.030s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":25330,"dirs.run_cpu_time_us":274,"dirs.run_wall_time_us":3144,"drs_written":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4,"lbm_write_time_us":949,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":25451,"threads_started":2,"wal-append.queue_time_us":373}
I20240502 14:28:56.011826 20673 maintenance_manager.cc:419] P a834edffbcb2469aae598b069aab0681: Scheduling UndoDeltaBlockGCOp(b13bcd6e693f4c7687a1aabdafbb83c6): 4925 bytes on disk
I20240502 14:28:56.012681 20563 maintenance_manager.cc:643] P a834edffbcb2469aae598b069aab0681: UndoDeltaBlockGCOp(b13bcd6e693f4c7687a1aabdafbb83c6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240502 14:28:56.047662 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:56.049082 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:56.051013 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:28:56.058430 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:56.061791 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:56.070086 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:56.078476 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:56.210651 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 14:28:56.219257 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 14:28:56.283345 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:28:56.338737 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:56.362811 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:56.370838 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:56.417996 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:56.441372 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:56.487717 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:56.488433 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:56.489557 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:56.490675 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:56.512217 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:28:56.563047 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:56.568715 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 14:28:56.570197 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:28:56.602687 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:56.606451 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:56.619921 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:56.652272 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:28:56.666080 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:56.796869 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:28:56.806648 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 14:28:56.826539 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:28:56.866408 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:56.945071 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:56.945971 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:28:56.998134 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:28:57.009764 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:28:57.025254 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:28:57.054754 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:57.071547 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:57.077371 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:28:57.085271 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:57.085795 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:57.086190 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:57.132009 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:57.161365 21077 maintenance_manager.cc:419] P ff6acba046404ef0a6f41ad5e09d852f: Scheduling FlushMRSOp(2c2afaffde96462eacd0d32bbdec23b1): perf score=0.033366
W20240502 14:28:57.167608 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:28:57.212918 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:28:57.233012 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:28:57.270275 20673 maintenance_manager.cc:419] P a834edffbcb2469aae598b069aab0681: Scheduling FlushMRSOp(db56136f2dbb429098d4d78b15a3a439): perf score=0.033341
W20240502 14:28:57.270895 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:28:57.279633 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:28:57.280159 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:57.316154 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:57.323415 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:57.335363 20965 maintenance_manager.cc:643] P ff6acba046404ef0a6f41ad5e09d852f: FlushMRSOp(2c2afaffde96462eacd0d32bbdec23b1) complete. Timing: real 0.174s user 0.072s sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":2029,"dirs.run_cpu_time_us":240,"dirs.run_wall_time_us":23169,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":997,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":6996,"threads_started":2,"wal-append.queue_time_us":4939}
I20240502 14:28:57.339732 21077 maintenance_manager.cc:419] P ff6acba046404ef0a6f41ad5e09d852f: Scheduling UndoDeltaBlockGCOp(2c2afaffde96462eacd0d32bbdec23b1): 4840 bytes on disk
I20240502 14:28:57.340711 20965 maintenance_manager.cc:643] P ff6acba046404ef0a6f41ad5e09d852f: UndoDeltaBlockGCOp(2c2afaffde96462eacd0d32bbdec23b1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4}
I20240502 14:28:57.347359 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:57.349853 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:57.365319 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:57.369853 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:57.392979 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:57.410454 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:28:57.458657 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:28:57.468447 20563 maintenance_manager.cc:643] P a834edffbcb2469aae598b069aab0681: FlushMRSOp(db56136f2dbb429098d4d78b15a3a439) complete. Timing: real 0.198s user 0.070s sys 0.059s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":62044,"dirs.run_cpu_time_us":242,"dirs.run_wall_time_us":2695,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":907,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":90279,"threads_started":2,"wal-append.queue_time_us":28423}
I20240502 14:28:57.469995 20673 maintenance_manager.cc:419] P a834edffbcb2469aae598b069aab0681: Scheduling UndoDeltaBlockGCOp(db56136f2dbb429098d4d78b15a3a439): 4586 bytes on disk
I20240502 14:28:57.471737 20563 maintenance_manager.cc:643] P a834edffbcb2469aae598b069aab0681: UndoDeltaBlockGCOp(db56136f2dbb429098d4d78b15a3a439) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
W20240502 14:28:57.509614 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:28:57.594477 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:57.618461 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:28:57.638469 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:28:57.640208 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:57.641628 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:57.645891 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:57.667269 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:57.679517 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:57.769147 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:57.849213 21077 maintenance_manager.cc:419] P ff6acba046404ef0a6f41ad5e09d852f: Scheduling FlushMRSOp(1ecc6539b85d40e49df7ffc5f93cb801): perf score=0.033370
I20240502 14:28:57.889077 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:28:57.892926 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:28:57.901643 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:57.928368 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:57.942602 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:57.950683 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:57.960234 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:28:57.996654 20965 maintenance_manager.cc:643] P ff6acba046404ef0a6f41ad5e09d852f: FlushMRSOp(1ecc6539b85d40e49df7ffc5f93cb801) complete. Timing: real 0.147s user 0.032s sys 0.034s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":32454,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":5734,"drs_written":1,"lbm_read_time_us":1189,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":1855,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":52766,"threads_started":2,"wal-append.queue_time_us":20277}
W20240502 14:28:57.998078 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:57.998200 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:57.999125 21077 maintenance_manager.cc:419] P ff6acba046404ef0a6f41ad5e09d852f: Scheduling UndoDeltaBlockGCOp(1ecc6539b85d40e49df7ffc5f93cb801): 4925 bytes on disk
I20240502 14:28:58.000589 20965 maintenance_manager.cc:643] P ff6acba046404ef0a6f41ad5e09d852f: UndoDeltaBlockGCOp(1ecc6539b85d40e49df7ffc5f93cb801) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
W20240502 14:28:58.092978 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:28:58.093307 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:28:58.131111 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:58.132299 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:58.158579 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:58.174007 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:58.179402 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:58.180290 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:58.238054 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:58.239486 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:28:58.288913 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:28:58.406451 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:58.427275 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:58.427556 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:58.433058 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:58.433866 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:58.441890 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:58.456468 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:28:58.535118 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:28:58.602136 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:28:58.604646 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:28:58.644539 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:28:58.710664 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:58.727857 20673 maintenance_manager.cc:419] P a834edffbcb2469aae598b069aab0681: Scheduling FlushMRSOp(6a992d12aee7416096d1353533397386): perf score=0.033382
I20240502 14:28:58.737871 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:58.740806 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:58.740782 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:58.797137 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:58.799317 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:58.812023 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:58.825910 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:58.872222 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:58.872200 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:58.891535 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:58.897799 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:28:58.936492 20563 maintenance_manager.cc:643] P a834edffbcb2469aae598b069aab0681: FlushMRSOp(6a992d12aee7416096d1353533397386) complete. Timing: real 0.208s user 0.058s sys 0.041s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":19486,"dirs.run_cpu_time_us":240,"dirs.run_wall_time_us":4523,"drs_written":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4,"lbm_write_time_us":984,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":106816,"threads_started":2,"wal-append.queue_time_us":84960}
I20240502 14:28:58.938227 20673 maintenance_manager.cc:419] P a834edffbcb2469aae598b069aab0681: Scheduling UndoDeltaBlockGCOp(6a992d12aee7416096d1353533397386): 4585 bytes on disk
I20240502 14:28:58.939043 20563 maintenance_manager.cc:643] P a834edffbcb2469aae598b069aab0681: UndoDeltaBlockGCOp(6a992d12aee7416096d1353533397386) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240502 14:28:58.949213 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:58.977208 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:58.986382 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:59.017727 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:28:59.119876 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:59.127197 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:59.144028 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:59.159605 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:59.160012 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:28:59.163372 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:59.186188 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:59.194710 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:28:59.196415 20673 maintenance_manager.cc:419] P a834edffbcb2469aae598b069aab0681: Scheduling FlushMRSOp(ecb8a9b04ba34f779e2227e77e6f764c): perf score=0.033391
W20240502 14:28:59.232952 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:28:59.305130 20563 maintenance_manager.cc:643] P a834edffbcb2469aae598b069aab0681: FlushMRSOp(ecb8a9b04ba34f779e2227e77e6f764c) complete. Timing: real 0.108s user 0.061s sys 0.014s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":77,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":1411,"drs_written":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4,"lbm_write_time_us":889,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":267,"threads_started":1,"wal-append.queue_time_us":429}
I20240502 14:28:59.307050 20673 maintenance_manager.cc:419] P a834edffbcb2469aae598b069aab0681: Scheduling UndoDeltaBlockGCOp(ecb8a9b04ba34f779e2227e77e6f764c): 4925 bytes on disk
I20240502 14:28:59.307816 20563 maintenance_manager.cc:643] P a834edffbcb2469aae598b069aab0681: UndoDeltaBlockGCOp(ecb8a9b04ba34f779e2227e77e6f764c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":133,"lbm_reads_lt_1ms":4}
W20240502 14:28:59.337891 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:28:59.361387 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:28:59.388662 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:28:59.389113 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:28:59.437100 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:28:59.480581 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:59.489991 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:59.511140 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:59.515460 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:59.539692 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:28:59.548943 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:59.579916 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:28:59.591848 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:28:59.596922 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:59.627909 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:28:59.632416 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:28:59.663951 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:59.665001 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:59.670384 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:59.688177 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:28:59.705670 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:28:59.746181 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:28:59.763283 21077 maintenance_manager.cc:419] P ff6acba046404ef0a6f41ad5e09d852f: Scheduling FlushMRSOp(db56136f2dbb429098d4d78b15a3a439): perf score=0.033371
W20240502 14:28:59.790704 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:28:59.810053 20965 maintenance_manager.cc:643] P ff6acba046404ef0a6f41ad5e09d852f: FlushMRSOp(db56136f2dbb429098d4d78b15a3a439) complete. Timing: real 0.047s user 0.027s sys 0.014s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1314,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":1884,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":831,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":1425,"threads_started":2,"wal-append.queue_time_us":311}
I20240502 14:28:59.811302 21077 maintenance_manager.cc:419] P ff6acba046404ef0a6f41ad5e09d852f: Scheduling UndoDeltaBlockGCOp(db56136f2dbb429098d4d78b15a3a439): 4586 bytes on disk
I20240502 14:28:59.811856 20965 maintenance_manager.cc:643] P ff6acba046404ef0a6f41ad5e09d852f: UndoDeltaBlockGCOp(db56136f2dbb429098d4d78b15a3a439) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
W20240502 14:28:59.855938 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:28:59.879681 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:28:59.899847 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:28:59.912026 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:28:59.954676 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:28:59.955516 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:28:59.971204 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:59.975467 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:28:59.977806 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:28:59.982873 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:00.155762 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:29:00.192677 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:00.207996 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:00.210201 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:00.209139 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:00.226121 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:00.233884 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:00.244498 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:29:00.250561 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:00.272634 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:29:00.308877 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:00.309590 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:29:00.329021 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:29:00.494349 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:00.497763 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:29:00.500119 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:00.501458 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:00.521342 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:00.526693 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:00.577963 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:00.591094 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:00.625519 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:00.684542 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:29:00.709245 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:00.709709 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:00.720827 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:00.727017 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:00.775846 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:29:00.776054 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:00.790320 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:00.852394 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:29:00.893203 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:29:00.973062 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:00.997756 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:01.024889 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:01.037417 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:01.054730 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:01.067766 20673 maintenance_manager.cc:419] P a834edffbcb2469aae598b069aab0681: Scheduling FlushMRSOp(51ef0be93c914e258635e13f3cca79aa): perf score=0.033388
I20240502 14:29:01.069089 21077 maintenance_manager.cc:419] P ff6acba046404ef0a6f41ad5e09d852f: Scheduling FlushMRSOp(51ef0be93c914e258635e13f3cca79aa): perf score=0.033349
I20240502 14:29:01.090706 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:01.095367 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:29:01.131341 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:01.135337 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:01.156435 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:01.193253 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:01.197098 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:01.213598 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:01.247795 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 14:29:01.250973 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:29:01.259930 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 91 similar messages]
I20240502 14:29:01.306234 20965 maintenance_manager.cc:643] P ff6acba046404ef0a6f41ad5e09d852f: FlushMRSOp(51ef0be93c914e258635e13f3cca79aa) complete. Timing: real 0.235s user 0.043s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6296,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":5932,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1048,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":15422,"threads_started":2,"wal-append.queue_time_us":1126}
I20240502 14:29:01.312085 21077 maintenance_manager.cc:419] P ff6acba046404ef0a6f41ad5e09d852f: Scheduling UndoDeltaBlockGCOp(51ef0be93c914e258635e13f3cca79aa): 4925 bytes on disk
I20240502 14:29:01.312657 20965 maintenance_manager.cc:643] P ff6acba046404ef0a6f41ad5e09d852f: UndoDeltaBlockGCOp(51ef0be93c914e258635e13f3cca79aa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
W20240502 14:29:01.335209 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:01.341724 20563 maintenance_manager.cc:643] P a834edffbcb2469aae598b069aab0681: FlushMRSOp(51ef0be93c914e258635e13f3cca79aa) complete. Timing: real 0.274s user 0.056s sys 0.023s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":9872,"dirs.run_cpu_time_us":235,"dirs.run_wall_time_us":9281,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1083,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":16623,"threads_started":2,"wal-append.queue_time_us":6673}
I20240502 14:29:01.343673 20673 maintenance_manager.cc:419] P a834edffbcb2469aae598b069aab0681: Scheduling FlushMRSOp(d28db16c9f7e405b8866f0e6c548e246): perf score=0.033457
W20240502 14:29:01.376278 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:01.392145 20563 maintenance_manager.cc:643] P a834edffbcb2469aae598b069aab0681: FlushMRSOp(d28db16c9f7e405b8866f0e6c548e246) complete. Timing: real 0.048s user 0.036s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":2083,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":712,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":6316,"threads_started":1,"wal-append.queue_time_us":6421}
I20240502 14:29:01.393573 20673 maintenance_manager.cc:419] P a834edffbcb2469aae598b069aab0681: Scheduling UndoDeltaBlockGCOp(51ef0be93c914e258635e13f3cca79aa): 4925 bytes on disk
I20240502 14:29:01.394184 20563 maintenance_manager.cc:643] P a834edffbcb2469aae598b069aab0681: UndoDeltaBlockGCOp(51ef0be93c914e258635e13f3cca79aa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240502 14:29:01.395184 20673 maintenance_manager.cc:419] P a834edffbcb2469aae598b069aab0681: Scheduling UndoDeltaBlockGCOp(d28db16c9f7e405b8866f0e6c548e246): 4923 bytes on disk
I20240502 14:29:01.395787 20563 maintenance_manager.cc:643] P a834edffbcb2469aae598b069aab0681: UndoDeltaBlockGCOp(d28db16c9f7e405b8866f0e6c548e246) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
W20240502 14:29:01.473760 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:29:01.657328 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 14:29:01.658584 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:29:01.687031 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:29:01.698683 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:01.704507 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:29:01.757393 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:01.757849 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:01.763006 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:29:01.787906 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:01.808387 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:01.813151 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:29:01.837320 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:29:01.845093 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:01.863246 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:01.905845 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:01.957422 19032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 14:29:01.969730 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:01.972534 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:01.986227 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:29:01.998907 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:02.001169 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:02.013741 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:02.021837 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:29:02.038471 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:02.041147 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:29:02.127537 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:29:02.186060 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:02.212621 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:29:02.248562 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:02.265794 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:29:02.280516 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:02.284901 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:02.302304 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:02.312171 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:02.329095 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:02.368721 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:02.384464 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:02.384840 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:02.411525 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:02.481880 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:02.503080 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:29:02.506150 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:02.512933 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:02.540920 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:29:02.581395 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:02.585824 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:29:02.653883 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:29:02.681492 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:29:02.692173 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:02.704205 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:02.711202 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:02.718535 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:02.738366 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:02.769078 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:02.802253 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:29:02.862658 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:02.871645 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:29:02.893667 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:29:02.941269 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:29:02.979439 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:02.986742 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:02.988898 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:02.989841 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:02.990170 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:02.992861 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:03.028450 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:29:03.051805 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:03.151758 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:29:03.228783 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:03.234545 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:29:03.253985 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:03.286932 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:03.300439 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:03.337875 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:03.339452 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:03.342304 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:03.353451 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:03.372395 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:29:03.374405 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:03.415848 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:03.457685 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:03.479419 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:03.487200 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:03.507736 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:29:03.585950 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:03.709554 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:03.716984 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:03.731645 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:03.732839 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:03.734756 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:03.734797 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:03.737707 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:03.846141 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:29:03.896281 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:29:03.993609 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:04.020004 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:04.032625 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:04.038121 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:04.044557 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:04.053421 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:04.054988 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:29:04.110193 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:04.127130 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:04.144071 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 14:29:04.151402 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:04.155107 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:04.181849 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:04.182010 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:04.186354 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:04.195722 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:29:04.226486 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:29:04.249989 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:04.266614 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:29:04.286131 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:29:04.378582 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:04.386865 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:29:04.387563 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:04.404732 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:04.415011 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:29:04.418553 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:04.444226 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:04.445446 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:04.489614 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:29:04.500298 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:29:04.545418 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:04.604365 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:04.656497 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:04.656900 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:04.669164 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:04.669507 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:04.686221 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:04.691314 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:29:04.695250 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:04.753883 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:29:04.778687 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:29:04.827356 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:29:04.835343 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:29:04.942052 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:04.955129 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:04.998790 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:05.004455 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:05.012722 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:05.027042 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:05.044906 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:05.054196 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:05.106000 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:29:05.139705 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:05.165326 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:29:05.171371 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:05.180168 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:05.181069 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:05.196425 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:29:05.198176 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:29:05.237843 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:05.241595 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:05.285198 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:29:05.292194 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:29:05.295507 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:29:05.437518 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:05.440843 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:29:05.448282 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:29:05.468197 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:29:05.475994 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:05.482053 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:05.490006 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:05.512713 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:05.515797 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:05.640765 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:05.654752 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:05.663355 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:05.673403 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:05.700291 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:05.704612 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:05.707755 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:05.708593 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:05.772598 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:29:05.880524 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:29:06.047783 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:06.076218 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:06.083199 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:06.085093 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:06.088917 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:06.101732 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:06.110554 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:06.177809 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:06.298475 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 82 similar messages]
I20240502 14:29:06.314321 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:06.334394 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:06.343091 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:06.353503 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 14:29:06.365243 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:29:06.370774 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:06.403620 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:06.408262 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:06.446532 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:29:06.453692 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:29:06.529352 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:29:06.599995 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:06.635880 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:06.639586 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:06.654052 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:06.674450 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:29:06.703267 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:29:06.705811 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:06.708878 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:06.712436 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:06.736197 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 14:29:06.737061 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:29:06.745828 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:06.765734 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:29:06.768803 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:29:06.831235 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:06.842866 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:06.863747 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:29:06.864526 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:06.865861 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:06.871426 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:06.890467 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:07.012629 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:29:07.013250 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:29:07.111732 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 14:29:07.131708 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:07.152746 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:07.164938 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:07.166085 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:07.196117 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:07.225239 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:07.268500 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:07.269467 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:29:07.277065 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:07.301143 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:29:07.304785 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:29:07.325134 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:07.353674 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:07.425530 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:07.426847 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:07.429003 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:07.439762 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:07.452896 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:29:07.455910 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:29:07.528190 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:29:07.648052 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:07.648823 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:29:07.651801 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:07.658421 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:07.669734 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:07.671067 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:07.680537 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:29:07.681977 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:29:07.691955 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:07.711323 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:07.772765 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:29:07.856797 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:29:07.861795 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:07.861915 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:07.871524 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:07.877135 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:07.883200 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:07.890547 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:07.904225 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:07.948771 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:29:07.978695 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:29:08.177832 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:08.191545 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:08.224102 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:08.224843 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:08.232895 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:08.234248 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:29:08.263933 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:08.279855 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:08.294703 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:08.325781 18637 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.67:45409, user_credentials={real_user=slave}} blocked reactor thread for 109281us
I20240502 14:29:08.326607 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:08.360849 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:08.400895 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:08.405122 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:08.417264 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:08.442327 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:29:08.445458 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:29:08.460229 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:08.686547 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:29:08.697118 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:08.710850 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:08.712260 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:08.729611 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:08.730616 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:08.736042 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:08.762909 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:08.778066 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:08.789425 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:08.815284 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:08.835870 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:08.868125 20562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 14:29:08.882309 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:08.894817 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:29:08.907653 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:08.937932 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:29:09.044371 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:09.084197 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:29:09.140664 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:29:09.154784 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:29:09.155925 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:29:09.157006 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:09.182024 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:29:09.224735 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:29:09.257483 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:29:09.340798 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:09.346329 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:09.346772 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:09.373296 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:29:09.386283 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:09.420132 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:09.427762 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:09.450878 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:09.451018 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:09.471979 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:09.491354 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:29:09.498821 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:29:09.554884 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:09.555982 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:09.574468 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:09.587852 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:09.678479 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:29:09.711211 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:09.750623 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:09.787215 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:29:09.796129 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:09.810981 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:29:09.830117 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:09.831281 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:09.857656 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:09.878268 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:29:09.881986 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:10.020236 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:10.028259 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:10.040817 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:10.048127 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:10.072762 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:10.072937 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:10.073858 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:10.126646 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:29:10.135187 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:29:10.139768 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:29:10.157164 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:29:10.197332 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:29:10.260828 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:10.335136 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:10.335739 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:10.351240 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:29:10.357090 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:10.368912 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:10.374305 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:10.391633 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:29:10.404335 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:10.420311 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:29:10.493455 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:29:10.545418 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:10.562906 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:10.562924 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:10.566329 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:10.592376 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:29:10.600039 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:10.600930 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:10.640031 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:10.813429 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:10.832048 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:29:10.832531 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:10.849572 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:10.883471 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:10.883723 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:10.890476 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:29:10.904870 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:10.905263 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:11.026090 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:11.033864 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:11.036900 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:11.039805 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:11.041179 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:11.041322 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:11.076352 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:11.242759 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:29:11.251361 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:29:11.255754 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:29:11.262933 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:11.292078 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:11.301054 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:11.322760 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 14:29:11.324025 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:29:11.324456 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:11.342031 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:11.345919 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:11.401515 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 14:29:11.424757 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:11.504722 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:11.516212 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:11.531636 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:29:11.532094 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:11.558923 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:11.563045 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:11.566198 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:11.629966 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:29:11.636133 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:11.762748 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:11.794325 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:11.797648 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 14:29:11.797973 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:29:11.804212 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:11.810860 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:29:11.829988 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:11.841265 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:11.856482 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:11.871037 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:11.922834 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:11.924983 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:11.950610 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:29:11.957442 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:11.964941 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:12.006673 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:29:12.008859 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:12.014539 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:12.115880 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:29:12.211894 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:12.251201 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:29:12.304194 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:12.313768 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:12.335259 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240502 14:29:12.337986 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:12.343358 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:29:12.345623 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:12.346352 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:12.350128 20561 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.72:35871, user_credentials={real_user=slave}} blocked reactor thread for 44004.8us
W20240502 14:29:12.365401 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:29:12.371968 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:29:12.373741 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:12.418861 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:29:12.443506 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:12.525576 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:29:12.549368 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:29:12.591609 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:29:12.615851 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:12.645643 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:12.665812 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:12.679834 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:12.688105 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:29:12.690800 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:29:12.699309 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:12.705607 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:12.712909 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:12.749403 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:29:12.794358 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:12.800487 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:12.820916 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:29:12.821128 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:12.824111 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:29:12.845574 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:29:12.859764 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:12.879660 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:12.893899 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:12.913936 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:12.974478 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:29:13.121575 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:13.129110 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:13.131691 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:13.161664 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:13.174942 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:13.183367 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:13.206116 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:13.235636 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:13.331259 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:13.335966 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:13.336818 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:13.344761 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:13.346210 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:13.346702 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:29:13.363101 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:13.367798 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:29:13.504899 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:29:13.532347 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:29:13.596626 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:13.646245 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:13.662432 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:13.662844 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:13.668605 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:29:13.681612 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:13.704377 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:13.720307 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:13.752285 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:13.824592 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:29:13.929354 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:29:13.938395 20562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 14:29:13.994508 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:14.015823 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:14.040839 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:14.043414 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:14.066890 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:14.071943 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:29:14.073215 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:14.102133 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:14.220616 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:14.226640 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:14.251873 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:14.296898 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:14.311105 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:14.312680 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:29:14.371832 20964 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.5.107.71:39195, user_credentials={real_user=slave}} blocked reactor thread for 59152us
W20240502 14:29:14.373324 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:14.379823 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:14.508764 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:29:14.514988 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:29:14.531220 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:14.541050 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:14.582720 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:14.583868 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:29:14.598949 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:14.618043 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:14.621495 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:14.641294 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:29:14.647707 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:14.697373 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:29:14.788013 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:14.795423 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:14.804008 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:14.808132 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:14.808465 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:14.809489 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:14.813676 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:14.844038 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:29:14.861210 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:29:14.909386 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:29:14.914953 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:29:14.973343 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:29:14.982033 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:29:15.029091 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:29:15.046859 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:15.056833 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:15.056990 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:15.062171 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:15.074977 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:15.083981 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:29:15.104852 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:15.139282 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:29:15.180305 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:15.337819 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:29:15.351953 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:29:15.372937 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:15.374703 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:29:15.385099 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:15.396226 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:15.396448 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:15.418844 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:15.431953 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:29:15.444162 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:15.460465 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:15.462742 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:15.489732 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:29:15.495419 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:15.529354 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:15.592294 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:29:15.611081 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:15.622469 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:15.647297 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:15.710952 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:15.806934 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:29:15.862255 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:15.866389 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:15.869083 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:15.889300 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:15.889288 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:15.896919 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:15.973767 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:16.115669 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:29:16.165866 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:29:16.167964 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:16.169541 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:16.176220 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:16.207336 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:16.207952 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:16.225853 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:16.244532 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:29:16.265447 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:29:16.274643 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:16.319942 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:29:16.349849 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:16.360379 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:16.368286 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:29:16.380170 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 81 similar messages]
I20240502 14:29:16.381633 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:16.405300 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:16.435420 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:16.446548 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:16.459546 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:16.473860 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 14:29:16.537454 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:29:16.636920 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:16.638888 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:16.642015 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:16.648463 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:16.655328 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:16.663269 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:16.712612 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:16.833904 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:16.882066 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:16.898411 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:16.898842 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:16.902259 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:16.903961 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:16.905536 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 57 similar messages]
I20240502 14:29:16.910949 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:16.952417 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:29:16.963860 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:29:17.056941 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:29:17.100980 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:29:17.101262 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:17.211858 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:17.232535 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:29:17.235003 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:17.248111 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:17.255244 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:17.276280 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:17.316972 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:17.317831 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:17.321377 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:29:17.340641 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:17.373777 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:29:17.392563 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:17.422662 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:17.423897 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:17.432627 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:29:17.442538 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:17.453683 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:17.490442 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:29:17.503281 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:29:17.536412 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:17.565960 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:29:17.570799 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:29:17.613633 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 14:29:17.615103 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:29:17.626555 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:29:17.750315 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:17.754487 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:29:17.768862 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:17.777793 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:17.785952 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:17.801148 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:17.826305 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:17.888257 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:17.896385 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:29:17.917039 20964 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.5.107.71:39195, user_credentials={real_user=slave}} blocked reactor thread for 51041.7us
W20240502 14:29:17.922969 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:17.923262 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:17.954427 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:17.959017 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:29:17.964171 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:17.985249 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:29:17.985980 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:18.001644 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:29:18.008880 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:18.016222 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:18.334676 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:29:18.362787 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:29:18.399935 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:18.428933 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:18.432446 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:18.467820 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:18.497041 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:18.500221 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:18.547709 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:18.598176 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:18.634550 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:29:18.711367 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:18.712291 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:18.744029 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:18.762060 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:18.762748 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:29:18.763432 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:18.765714 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:18.850677 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:18.880960 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:29:18.881485 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:29:18.881913 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:18.896121 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:29:18.961308 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:29:18.968019 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:18.988938 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:19.005084 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:19.009905 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:19.015494 20562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 14:29:19.020191 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:19.030300 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:19.096138 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:29:19.188930 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:19.194111 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:19.215927 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:19.219512 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:19.231443 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:19.269654 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:19.441900 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:19.478595 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 14:29:19.488078 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:19.490980 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:29:19.509793 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:19.516587 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:19.517649 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:19.518187 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:19.521580 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:19.552927 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:19.556800 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:29:19.622869 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:29:19.702551 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:29:19.711445 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:19.712945 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:19.723661 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:19.734190 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:19.743531 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:19.750152 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:19.794363 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:29:19.795497 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:29:19.894392 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:29:19.905223 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:29:19.938396 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:29:19.953285 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:19.965596 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:29:19.985729 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:19.993587 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:19.997272 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:20.002337 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:20.012980 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:20.016062 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:20.016311 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:20.059613 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:29:20.204572 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:20.212503 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:20.229089 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:20.230628 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:20.234254 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:20.238180 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:29:20.252185 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:29:20.258143 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:20.273485 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:29:20.301513 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:29:20.307641 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:29:20.383160 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:20.466398 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:20.473356 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:29:20.474524 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:20.490500 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:20.493206 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:20.512476 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:20.553467 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:20.623627 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:29:20.637050 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:20.653375 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:29:20.689486 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:20.691147 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:20.707748 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:20.708300 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:20.727031 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:20.774273 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:20.801546 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:20.838752 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:29:20.905323 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:20.910797 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:20.944381 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:20.944684 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:29:20.973573 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:20.987231 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:21.024258 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:21.135210 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:29:21.148393 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:21.170292 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:21.171963 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:21.184679 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:21.186014 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:21.187983 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:21.188041 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:21.206451 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:21.300462 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:29:21.332957 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:29:21.378831 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:29:21.406945 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:29:21.451133 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:21.455119 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:21.467368 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 14:29:21.474603 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 90 similar messages]
I20240502 14:29:21.477293 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:21.487318 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:21.497412 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:21.549456 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:21.597985 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:29:21.626441 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:29:21.689110 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:21.787034 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:21.846482 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:21.873420 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:21.890602 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:21.904618 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:21.919076 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:21.930392 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240502 14:29:21.938051 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:21.942482 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:21.964937 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:21.972645 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:29:21.988373 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:22.002972 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:22.010253 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:22.017561 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:22.036680 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:29:22.088513 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:29:22.117753 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:29:22.141614 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:29:22.228837 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:22.281601 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:29:22.306185 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:22.340433 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:22.343621 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:22.353003 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:22.356009 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:22.359050 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:22.369715 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:29:22.370105 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:29:22.371074 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:22.377540 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:22.402642 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:22.424914 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:22.429553 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:29:22.471057 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:22.481655 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:22.553606 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:22.558261 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:29:22.658351 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:29:22.780027 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 14:29:22.780917 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:29:22.788854 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:29:22.807873 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:22.828243 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:22.854048 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:29:22.922169 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:29:22.938309 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:29:22.968453 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:29:23.009896 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:29:23.016168 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:23.041110 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:23.058837 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:23.071267 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:23.077987 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:23.090469 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:23.094998 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:23.100170 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:23.162762 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:29:23.180640 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:23.201931 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:23.204931 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:23.217481 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:23.250139 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:23.253904 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:29:23.299423 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:23.430039 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:23.447209 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:23.473969 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:23.485877 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:23.555078 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:23.561312 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:29:23.576927 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:29:23.587584 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:23.629653 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:23.648489 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:29:23.689260 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:29:23.707545 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:23.711607 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:23.727896 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:23.731062 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:23.732174 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:23.745971 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:23.759699 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:23.760737 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:29:23.765794 20673 maintenance_manager.cc:419] P a834edffbcb2469aae598b069aab0681: Scheduling FlushMRSOp(21af917743444f2db932636a78f33512): perf score=0.033360
W20240502 14:29:23.790750 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:29:23.848783 20563 maintenance_manager.cc:643] P a834edffbcb2469aae598b069aab0681: FlushMRSOp(21af917743444f2db932636a78f33512) complete. Timing: real 0.083s user 0.047s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":1205,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":950,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":818,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":3289,"threads_started":2,"wal-append.queue_time_us":2262}
I20240502 14:29:23.849975 20673 maintenance_manager.cc:419] P a834edffbcb2469aae598b069aab0681: Scheduling UndoDeltaBlockGCOp(21af917743444f2db932636a78f33512): 4840 bytes on disk
I20240502 14:29:23.850594 20563 maintenance_manager.cc:643] P a834edffbcb2469aae598b069aab0681: UndoDeltaBlockGCOp(21af917743444f2db932636a78f33512) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240502 14:29:23.985363 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:23.996296 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:24.013727 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:24.040747 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:24.048249 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:24.074021 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:24.078534 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:29:24.111385 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:24.122144 20562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:29:24.239748 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:24.265664 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:24.291874 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:24.295883 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:24.297816 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:24.300033 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:24.311015 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:24.355077 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:29:24.518987 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:29:24.549887 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:24.560026 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:24.613950 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:24.617158 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:29:24.620862 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:24.622509 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:24.641016 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:29:24.658428 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:24.668361 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:24.671167 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:24.674046 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:24.683869 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:24.686692 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:24.701051 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:24.713969 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:24.749691 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:29:24.766250 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:24.782725 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:29:24.806643 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:29:24.839011 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:29:24.895035 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:29:24.904362 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:29:24.962146 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:24.978086 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:24.981690 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:24.983943 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:24.990825 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:24.994993 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:25.049315 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:29:25.061085 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:29:25.066828 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:29:25.155591 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:25.218847 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:29:25.236369 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:25.318836 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:25.348379 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:29:25.350590 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:25.369993 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:25.380800 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:29:25.395404 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:25.398921 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:29:25.407661 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:25.411016 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:25.433883 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:25.460949 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:25.462874 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:25.522749 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:25.546283 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:25.551096 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:25.620611 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:29:25.657950 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:29:25.662081 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:25.712440 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:29:25.744372 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:25.760905 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:25.767385 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:29:25.768270 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:29:25.769821 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:25.770021 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:25.787317 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:25.793780 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:25.795840 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:25.872347 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:29:26.012802 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:26.017591 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:26.033130 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:26.034407 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:26.057598 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:26.081923 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:26.166729 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:29:26.221253 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:26.246479 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:29:26.255393 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:26.256165 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:29:26.305438 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:26.310146 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:29:26.311029 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:26.334159 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:26.341846 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:26.355657 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:26.395180 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:26.423512 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:29:26.448318 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:26.465533 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:26.482717 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:26.500893 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:26.502180 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:26.506875 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:26.510129 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:29:26.517370 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240502 14:29:26.522141 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240502 14:29:26.629040 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:26.811218 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:26.815642 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:26.818010 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:26.818346 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:26.818794 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:29:26.832666 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:26.836979 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:26.860865 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:26.909632 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:29:26.952279 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:26.988817 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 14:29:26.990195 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:29:26.991650 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:26.999369 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:29:27.008617 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:27.010794 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:27.016953 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:27.026186 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:27.116410 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:29:27.152657 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:27.266044 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:29:27.270936 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:29:27.286976 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:27.421911 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:27.422863 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:27.453850 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:27.460086 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:27.485661 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:27.511516 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:29:27.521549 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:27.528391 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:29:27.545766 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:29:27.546097 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:29:27.607156 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:29:27.608574 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:29:27.627418 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:29:27.649526 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:27.682845 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:27.687027 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:27.708016 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:27.709069 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:27.725790 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:27.729307 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:27.772042 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:27.884267 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:27.924556 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:27.935017 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 14:29:27.940084 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:27.942366 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:27.961824 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:27.988744 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:28.004891 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:29:28.144716 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:28.144855 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:28.157760 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:28.161341 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:28.176795 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:28.176965 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:28.206243 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:28.210547 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:29:28.214040 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:29:28.224103 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:29:28.265069 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:29:28.265157 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:28.266265 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:29:28.334697 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:29:28.356554 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:29:28.530052 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:28.546142 19076 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:28.546584 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:28.550774 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:28.555137 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:28.571097 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:28.592839 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:28.598591 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:29:28.623169 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:28.627578 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:29:28.630102 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:28.641798 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:28.649729 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:29:28.683444 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:28.693377 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:28.710903 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:28.713076 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:28.914928 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:29:28.956943 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:29:28.958379 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:28.968154 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:28.978338 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:28.978710 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:29.001194 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:29.014482 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:29.071242 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:29.160576 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:29:29.214512 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:29.242480 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:29.252039 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:29.259131 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:29.265280 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:29.270702 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:29.273491 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:29:29.285727 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:29.287817 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:29:29.344591 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:29:29.405488 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:29:29.477522 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:29:29.491580 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:29:29.491652 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:29:29.494988 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:29.519017 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:29.552476 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:29.557458 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:29.570730 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:29.588191 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:29.616771 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:29.736507 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:29.740470 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:29.767091 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:29.768496 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:29.796757 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:29.819593 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:29:29.822094 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:29.828158 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:29.945704 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:29:30.033710 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:30.064723 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:29:30.070106 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:30.080277 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:29:30.081501 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:30.084569 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:29:30.096824 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:29:30.097159 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:30.110879 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:30.132469 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:30.137691 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:30.147523 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:29:30.150027 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:29:30.232306 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:29:30.307497 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:30.312434 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:30.326682 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:30.329815 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:30.330118 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:30.341926 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:30.372803 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:30.399442 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:29:30.520319 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:29:30.590648 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:30.603729 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:30.625936 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:30.640472 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:29:30.659106 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:30.674885 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:30.686410 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:30.690672 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:30.810334 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:29:30.816956 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:29:30.845484 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:29:30.910478 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:30.920338 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:30.927562 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:30.955578 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:29:30.956064 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:30.996765 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:29:31.004597 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:31.014614 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:31.044857 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:31.052258 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:31.058964 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:31.092552 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:31.107614 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:29:31.127413 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:31.147620 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:31.166085 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:31.193682 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:29:31.209447 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:31.252415 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:29:31.310591 19143 maintenance_manager.cc:419] P 007b2e79dc424a6f80141180f072f2d8: Scheduling FlushMRSOp(d28db16c9f7e405b8866f0e6c548e246): perf score=0.033392
W20240502 14:29:31.337076 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:29:31.352965 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:31.353125 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:31.370046 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:31.371717 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:31.376422 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:31.384650 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:31.400132 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:31.468142 19033 maintenance_manager.cc:643] P 007b2e79dc424a6f80141180f072f2d8: FlushMRSOp(d28db16c9f7e405b8866f0e6c548e246) complete. Timing: real 0.156s user 0.027s sys 0.024s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":23088,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":4119,"drs_written":1,"lbm_read_time_us":143,"lbm_reads_lt_1ms":4,"lbm_write_time_us":889,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":71469,"threads_started":2,"wal-append.queue_time_us":70252}
I20240502 14:29:31.469396 19143 maintenance_manager.cc:419] P 007b2e79dc424a6f80141180f072f2d8: Scheduling UndoDeltaBlockGCOp(d28db16c9f7e405b8866f0e6c548e246): 4923 bytes on disk
I20240502 14:29:31.471747 19033 maintenance_manager.cc:643] P 007b2e79dc424a6f80141180f072f2d8: UndoDeltaBlockGCOp(d28db16c9f7e405b8866f0e6c548e246) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
W20240502 14:29:31.530256 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 14:29:31.535427 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:29:31.568897 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 14:29:31.649912 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:29:31.681591 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:31.682377 18748 maintenance_manager.cc:419] P 2db304b538e14857b7b1cc272d0db94b: Scheduling FlushMRSOp(6a992d12aee7416096d1353533397386): perf score=0.033337
I20240502 14:29:31.688791 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:31.691759 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:29:31.725451 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:31.730180 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:31.746421 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:31.765587 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:31.767000 18638 maintenance_manager.cc:643] P 2db304b538e14857b7b1cc272d0db94b: FlushMRSOp(6a992d12aee7416096d1353533397386) complete. Timing: real 0.084s user 0.037s sys 0.017s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":9110,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":3652,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1720,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":254208,"thread_start_us":24066,"threads_started":2,"wal-append.queue_time_us":6546}
I20240502 14:29:31.768246 18748 maintenance_manager.cc:419] P 2db304b538e14857b7b1cc272d0db94b: Scheduling UndoDeltaBlockGCOp(6a992d12aee7416096d1353533397386): 4585 bytes on disk
I20240502 14:29:31.768934 18638 maintenance_manager.cc:643] P 2db304b538e14857b7b1cc272d0db94b: UndoDeltaBlockGCOp(6a992d12aee7416096d1353533397386) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4}
W20240502 14:29:31.769825 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:29:31.773404 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:31.801160 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:29:31.835518 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:29:31.842726 18637 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.67:45409, user_credentials={real_user=slave}} blocked reactor thread for 148597us
W20240502 14:29:31.878958 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:31.896054 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:31.932917 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:31.988550 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:31.989924 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:31.997911 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 14:29:32.002161 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:29:32.022390 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:32.031761 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:32.064496 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:29:32.074576 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:29:32.136863 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:32.182428 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:32.187073 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:32.191226 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:32.195399 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:32.208092 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:32.211433 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:32.423367 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:32.426983 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:32.435639 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:32.437134 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:32.458699 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:32.459846 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:32.484953 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:32.497550 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:29:32.534835 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:29:32.556228 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:29:32.600631 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:29:32.625313 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:29:32.711551 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:32.711998 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:29:32.729763 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:32.734791 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:32.769465 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:32.769955 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:32.790423 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:32.794927 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:32.930152 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:29:32.941715 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 14:29:32.957857 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:29:32.970481 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:29:32.995913 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:33.003065 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:33.007274 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:33.009565 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:33.013134 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:33.023025 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:33.033492 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:33.033632 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:29:33.161106 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:33.182106 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:29:33.194011 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:29:33.252385 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:33.265117 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:33.283666 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:33.289218 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:33.305632 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:33.311125 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:33.368281 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:29:33.372409 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:29:33.383030 18880 maintenance_manager.cc:419] P fec04c50dc5b49878ffa3c3873df32e4: Scheduling FlushMRSOp(db56136f2dbb429098d4d78b15a3a439): perf score=0.033335
W20240502 14:29:33.445878 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:33.483146 18770 maintenance_manager.cc:643] P fec04c50dc5b49878ffa3c3873df32e4: FlushMRSOp(db56136f2dbb429098d4d78b15a3a439) complete. Timing: real 0.100s user 0.041s sys 0.033s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":45142,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1476,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":845,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":45272,"threads_started":2,"wal-append.queue_time_us":338}
I20240502 14:29:33.484761 18880 maintenance_manager.cc:419] P fec04c50dc5b49878ffa3c3873df32e4: Scheduling UndoDeltaBlockGCOp(db56136f2dbb429098d4d78b15a3a439): 4586 bytes on disk
I20240502 14:29:33.485714 18770 maintenance_manager.cc:643] P fec04c50dc5b49878ffa3c3873df32e4: UndoDeltaBlockGCOp(db56136f2dbb429098d4d78b15a3a439) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":227,"lbm_reads_lt_1ms":4}
I20240502 14:29:33.518709 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:33.522192 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:33.530448 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:29:33.546473 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:33.566501 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:33.572518 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:33.588794 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:33.668172 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:29:33.715857 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:29:33.781302 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:33.788103 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:29:33.838706 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:33.843977 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:33.852320 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:33.854055 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:33.865514 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:33.883280 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:33.927945 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:33.988322 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:29:34.009812 19012 maintenance_manager.cc:419] P c902805060df4df9b5599d0e94b2dc06: Scheduling FlushMRSOp(0ffb6e582ef04fb38dd273c13febac58): perf score=0.033355
W20240502 14:29:34.045934 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:29:34.083269 18901 maintenance_manager.cc:643] P c902805060df4df9b5599d0e94b2dc06: FlushMRSOp(0ffb6e582ef04fb38dd273c13febac58) complete. Timing: real 0.073s user 0.041s sys 0.017s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":13545,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":1525,"drs_written":1,"lbm_read_time_us":151,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1010,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":21163,"threads_started":2,"wal-append.queue_time_us":7861}
I20240502 14:29:34.085093 19012 maintenance_manager.cc:419] P c902805060df4df9b5599d0e94b2dc06: Scheduling UndoDeltaBlockGCOp(0ffb6e582ef04fb38dd273c13febac58): 4840 bytes on disk
I20240502 14:29:34.085847 18901 maintenance_manager.cc:643] P c902805060df4df9b5599d0e94b2dc06: UndoDeltaBlockGCOp(0ffb6e582ef04fb38dd273c13febac58) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":138,"lbm_reads_lt_1ms":4}
I20240502 14:29:34.101811 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:34.121867 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:34.131873 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:34.133520 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:34.150987 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:34.172899 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:34.211611 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:29:34.260797 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:29:34.283427 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:29:34.301975 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:34.305696 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:34.313970 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:34.315410 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:29:34.332616 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:29:34.333266 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:29:34.336812 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:34.338454 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:34.344344 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:29:34.353271 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:34.374076 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:34.390815 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:34.462707 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:29:34.561295 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:34.570544 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:34.576016 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:34.585556 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:34.591063 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:34.603200 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:34.710240 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:29:34.741919 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:29:34.782864 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:34.866773 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:34.879204 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:29:34.886516 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:34.907101 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:34.907485 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:34.924573 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:29:34.931983 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:34.955138 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:34.966662 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:34.970417 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:34.985466 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:35.018299 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:35.019781 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:35.024945 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:29:35.044593 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:35.054525 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:35.123958 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:29:35.124332 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:29:35.266373 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:29:35.267345 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:35.308818 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:35.324759 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:35.328264 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:35.340211 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:35.341918 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:35.366226 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:35.374488 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:29:35.396777 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:29:35.463625 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:35.532498 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:29:35.547912 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:29:35.554059 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:29:35.586110 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:35.588232 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:35.593825 19012 maintenance_manager.cc:419] P c902805060df4df9b5599d0e94b2dc06: Scheduling FlushMRSOp(df5f14206a1d46e4866f34fa234f945b): perf score=0.033377
I20240502 14:29:35.599529 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:35.609970 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:35.616880 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:35.624540 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:35.661908 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:35.673488 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:29:35.709535 18901 maintenance_manager.cc:643] P c902805060df4df9b5599d0e94b2dc06: FlushMRSOp(df5f14206a1d46e4866f34fa234f945b) complete. Timing: real 0.115s user 0.031s sys 0.011s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":14364,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":3195,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":804,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":21746,"threads_started":2,"wal-append.queue_time_us":359}
I20240502 14:29:35.711077 19012 maintenance_manager.cc:419] P c902805060df4df9b5599d0e94b2dc06: Scheduling UndoDeltaBlockGCOp(df5f14206a1d46e4866f34fa234f945b): 4838 bytes on disk
I20240502 14:29:35.711987 18901 maintenance_manager.cc:643] P c902805060df4df9b5599d0e94b2dc06: UndoDeltaBlockGCOp(df5f14206a1d46e4866f34fa234f945b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
W20240502 14:29:35.841724 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:29:35.867882 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:35.873881 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:35.874807 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:29:35.903694 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:35.912220 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:35.914597 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:35.922717 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:35.938699 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:36.006574 21077 maintenance_manager.cc:419] P ff6acba046404ef0a6f41ad5e09d852f: Scheduling FlushMRSOp(097eb705c9e24740a5be61bf3354f3d8): perf score=0.033393
I20240502 14:29:36.026052 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:36.042322 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:29:36.045571 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:36.074694 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:36.097656 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:36.114168 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:36.149691 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:36.149860 20965 maintenance_manager.cc:643] P ff6acba046404ef0a6f41ad5e09d852f: FlushMRSOp(097eb705c9e24740a5be61bf3354f3d8) complete. Timing: real 0.143s user 0.055s sys 0.030s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":34417,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":2415,"drs_written":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1101,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":44496,"threads_started":2,"wal-append.queue_time_us":10067}
I20240502 14:29:36.150216 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:36.153213 21077 maintenance_manager.cc:419] P ff6acba046404ef0a6f41ad5e09d852f: Scheduling UndoDeltaBlockGCOp(097eb705c9e24740a5be61bf3354f3d8): 4586 bytes on disk
I20240502 14:29:36.155378 20965 maintenance_manager.cc:643] P ff6acba046404ef0a6f41ad5e09d852f: UndoDeltaBlockGCOp(097eb705c9e24740a5be61bf3354f3d8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
W20240502 14:29:36.209368 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:29:36.210798 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:29:36.254171 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:29:36.334831 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:36.346050 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:36.349876 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:36.376260 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:36.393437 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:36.401726 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:36.459620 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:36.526710 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:36.550359 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:36.552457 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:36.565163 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:36.568464 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:36.571390 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:36.584028 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240502 14:29:36.584648 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240502 14:29:36.585491 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:36.610930 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:29:36.628103 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:29:36.628180 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:29:36.719837 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:29:36.744576 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:29:36.777477 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:29:36.819182 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:36.833598 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:36.834558 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:36.846580 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:36.846671 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:36.864859 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:36.984640 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:29:36.984951 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:36.987881 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:29:37.014762 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:29:37.023015 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:29:37.082315 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:37.138224 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:37.139894 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:37.144809 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:37.157378 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:37.162267 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:37.182837 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:37.203523 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 14:29:37.250954 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:29:37.253787 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:29:37.372525 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:37.386827 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:37.402820 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:37.413257 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:37.414487 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:29:37.424752 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:37.439989 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:37.477483 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:37.507488 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:37.540302 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:29:37.600389 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:37.611636 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:37.626194 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:29:37.645949 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:37.647420 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:37.648856 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:37.654140 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:37.742890 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:29:37.743716 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:29:37.786795 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:29:37.872597 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:37.874025 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:37.886279 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:37.899881 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:37.906637 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:37.909466 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:37.932582 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:38.039368 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:38.059870 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:38.075932 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:38.077263 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:29:38.089948 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:29:38.123373 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:38.125789 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:38.126962 19032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 14:29:38.144593 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:29:38.150549 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:38.151607 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:38.241524 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:29:38.327301 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:38.334612 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:38.337677 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:38.344388 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:38.344938 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:38.356959 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:38.370874 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:29:38.460938 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:38.461233 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:29:38.545892 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:38.547984 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:29:38.565382 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:29:38.578819 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:38.603389 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:38.603529 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:38.606055 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:38.608896 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:38.614027 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:38.752156 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:29:38.827073 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:29:38.915103 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:38.933807 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:38.935729 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:38.938568 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:38.948071 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:38.955602 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:38.961231 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:38.962816 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:38.967294 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:39.046087 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:39.060169 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:39.106037 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:29:39.126521 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:39.133435 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:29:39.144667 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:39.169729 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:39.255223 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:29:39.291263 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:29:39.295141 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:39.295909 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:29:39.358990 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:29:39.374219 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:39.374230 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 14:29:39.397219 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:39.413460 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:39.413538 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:39.448861 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:39.463609 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:39.469364 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:29:39.487838 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:39.491966 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:39.492440 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:39.495231 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:39.496126 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:39.518501 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:29:39.557904 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:39.688259 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:29:39.696799 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:39.707072 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:29:39.807047 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:29:39.840250 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:39.853844 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:29:39.916029 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:29:40.004505 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:29:40.069615 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:40.096009 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:40.096101 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:40.121381 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:29:40.130829 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:40.133809 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:40.169898 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:40.170430 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:40.187672 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:29:40.208979 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:40.238293 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:40.263909 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:40.264165 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:40.271989 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:40.294015 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:40.306779 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:40.380558 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:29:40.389627 19031 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.72:35871, user_credentials={real_user=slave}} blocked reactor thread for 51991.9us
W20240502 14:29:40.506830 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:29:40.527887 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:29:40.540951 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:29:40.575408 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:29:40.577206 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:40.587127 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:40.608073 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:40.613148 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:29:40.614588 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:40.623845 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:29:40.633795 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:40.634361 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:40.635110 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:40.640213 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:40.683844 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:40.694205 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:40.698689 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:40.713980 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:40.730232 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:40.842558 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:40.855321 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:29:40.866650 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:29:41.000344 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:41.001881 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:41.002663 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:41.004225 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:41.011694 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:41.017788 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:41.157311 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:29:41.168641 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:41.232586 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:29:41.250968 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:41.263453 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:41.274092 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:29:41.283233 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:41.285776 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:41.288614 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:41.311791 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:41.435842 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:29:41.437145 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:29:41.445792 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:41.566815 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:41.580160 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:41.598163 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:41.631665 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:41.639196 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:41.644044 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 81 similar messages]
I20240502 14:29:41.644272 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:41.645229 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:29:41.645915 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:41.647635 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:41.684520 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 14:29:41.687741 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:29:41.702620 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:41.718932 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:41.733404 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:41.736655 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:29:41.737687 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:29:41.743270 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:29:41.750766 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:41.797547 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:41.998369 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:42.041565 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:42.046929 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:42.049535 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:42.050029 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:42.062453 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:42.090845 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:42.098506 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:29:42.185261 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:29:42.187629 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:29:42.207814 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:29:42.238591 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:42.295403 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 14:29:42.369478 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:29:42.371260 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:29:42.436760 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:29:42.440481 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:42.469681 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:42.497836 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:42.501340 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:42.507891 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:29:42.516165 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:42.519253 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:42.522935 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:42.548265 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:42.590972 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:42.617393 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:29:42.634594 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:42.635197 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:42.677631 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:42.679960 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:42.697921 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:42.797538 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:29:42.877530 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:29:42.879151 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:42.908959 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:42.910758 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:42.914466 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:42.918529 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:42.930788 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:43.009687 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:29:43.018215 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:29:43.030630 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:43.057577 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:43.127277 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:43.134264 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:43.137610 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:43.146543 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:43.163393 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:43.172626 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 14:29:43.185462 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:43.197324 18769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.69:41119, user_credentials={real_user=slave}} blocked reactor thread for 43565.6us
W20240502 14:29:43.213306 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:29:43.218083 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:29:43.307130 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:29:43.334450 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:29:43.466001 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:29:43.507232 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:43.523370 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:29:43.526077 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:43.530282 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:43.537051 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:43.537652 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:43.538653 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:43.541743 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:43.557677 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:43.596473 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:43.597860 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:43.619177 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:29:43.673118 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:29:43.713419 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:43.721961 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:43.722400 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:43.733927 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:43.968288 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:29:43.971259 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:43.978562 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:29:43.980584 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:43.999521 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:44.007784 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:29:44.017290 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:44.052924 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:44.089300 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:44.113874 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:29:44.119740 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:44.144636 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:44.158910 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:29:44.199612 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:44.201829 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:44.207155 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:44.232039 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:44.232617 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:44.234197 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:29:44.267167 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:44.365134 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:29:44.435593 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:44.520454 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:44.543012 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:44.546016 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:44.559119 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:44.559432 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 14:29:44.576367 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:44.595701 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:29:44.601241 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:44.609366 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:44.655812 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:29:44.766460 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:29:44.774550 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:29:44.787951 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:29:44.834659 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:44.846357 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:44.863114 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:44.882791 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:44.889688 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:44.900368 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:29:44.900868 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:44.915760 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:45.026641 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:45.048023 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:45.058861 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:29:45.082022 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:29:45.090288 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:45.111282 18769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.69:41119, user_credentials={real_user=slave}} blocked reactor thread for 43494.3us
I20240502 14:29:45.113673 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:45.118678 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:45.118835 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:29:45.163827 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:45.164139 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:45.168467 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:29:45.344465 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:45.348986 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:45.368466 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:45.371721 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:45.382126 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:45.388511 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:45.458606 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:45.487388 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:29:45.499625 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:29:45.522590 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:29:45.549850 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:29:45.571810 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:45.579273 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:29:45.580562 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:45.600224 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:45.601869 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:45.605054 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:45.615334 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:45.624161 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:45.728317 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:29:45.819833 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:29:45.824021 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:45.842726 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:45.847589 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:45.849058 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:45.858088 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:45.866461 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:45.919844 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:29:45.921310 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:45.954829 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:29:46.055912 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:46.136487 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:29:46.142434 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:46.168030 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:46.206507 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:46.219409 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:46.220077 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:46.230643 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:29:46.246692 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:46.285734 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:46.285861 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:29:46.303404 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:46.329366 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:29:46.373332 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:46.391862 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:46.414122 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:46.416437 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:46.439283 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:46.460716 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:29:46.474134 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:29:46.572479 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:29:46.597627 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:29:46.602934 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:29:46.604375 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:46.636268 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:46.663555 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:46.666249 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:46.668313 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:46.668946 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:46.693493 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:46.695152 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 14:29:46.812084 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 14:29:46.839637 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:46.926971 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:29:46.964745 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:46.997146 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:29:46.998551 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:47.006868 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:47.056250 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:47.075513 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:47.079115 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:47.144273 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:29:47.186709 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:47.197422 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:47.233172 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:47.247998 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:47.249756 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:29:47.263636 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:47.265650 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:47.267215 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:47.270617 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:29:47.322429 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:29:47.344673 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:47.345847 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 14:29:47.478531 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:47.483515 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:29:47.490689 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:47.503330 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:47.513799 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:47.535948 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:47.535974 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:29:47.536938 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:29:47.556272 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:47.640873 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:29:47.641105 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:29:47.679772 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:47.761508 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:47.790737 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:47.794137 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:47.805902 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:47.806866 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:47.808960 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:47.866621 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:29:47.869457 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:29:47.890069 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:47.924328 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:29:47.935657 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:29:48.077720 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:48.108950 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:48.128427 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:48.130677 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:48.148496 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:48.166754 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:48.167147 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:48.175724 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:48.187353 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 14:29:48.188232 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:29:48.191980 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:29:48.219141 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:48.231786 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:48.268134 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:29:48.271087 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:48.302791 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:48.349761 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:48.376147 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:48.471812 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:48.518992 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:29:48.555930 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:48.569434 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:48.578953 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:48.595904 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:29:48.597524 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:48.597983 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:48.632418 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:48.672273 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:48.718914 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:29:48.774205 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:48.785959 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:48.798089 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:48.814982 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:48.831627 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:48.833225 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:29:48.835573 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:48.840685 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:29:48.900548 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:29:48.901527 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:29:48.958531 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:29:49.029150 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:49.041581 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:29:49.067781 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:29:49.075973 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:49.084659 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:49.085429 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:49.086691 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:49.096880 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:49.099975 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:49.164682 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:49.395412 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:49.411041 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:49.446871 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:49.453496 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:49.478368 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:49.484226 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:49.508580 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:29:49.585906 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:29:49.599293 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:49.604380 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:49.612599 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:49.641031 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:49.645843 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:29:49.656616 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:29:49.658385 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:49.661450 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:49.678972 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:49.695315 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:29:49.723712 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:29:49.774850 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:49.789242 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:29:49.860136 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:49.861054 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:49.877969 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:49.880923 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:49.895557 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:49.901635 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:50.064072 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:29:50.132962 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:29:50.176918 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:50.177265 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:29:50.190454 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:50.201514 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:50.211325 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:50.213048 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:29:50.219245 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:50.236073 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:50.242030 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:50.255821 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:50.255897 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:50.296475 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:29:50.302368 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:29:50.311869 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:50.326190 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:50.363447 20964 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.5.107.71:39195, user_credentials={real_user=slave}} blocked reactor thread for 65397us
W20240502 14:29:50.364816 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:50.380717 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:50.435878 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:50.457505 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:29:50.484419 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:29:50.529484 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:29:50.654402 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:50.664101 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:50.675925 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:50.676580 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:50.677227 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:50.678792 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:29:50.692119 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:50.693763 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:50.706311 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:29:50.715917 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:29:50.880360 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:50.891381 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:29:50.900657 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:50.921290 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:50.927328 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:29:50.930240 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:50.947044 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:50.947959 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:50.966429 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:51.102267 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:51.167479 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:29:51.186079 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:51.203015 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:51.249828 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:51.253809 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:51.266947 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:29:51.281253 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:51.283665 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:51.291776 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:51.293813 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:51.335950 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:51.337302 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:51.355365 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:29:51.356971 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:29:51.392602 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:51.413563 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:29:51.414999 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:51.460441 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:29:51.471575 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:29:51.486065 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:51.488744 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:29:51.674919 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:51.681432 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:29:51.722754 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240502 14:29:51.724256 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:29:51.733156 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:51.753728 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:51.765731 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:51.792799 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:51.794984 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:51.802903 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:51.804435 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:51.838091 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:51.876319 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:29:51.882411 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 82 similar messages]
I20240502 14:29:51.910647 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:51.914402 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:51.943681 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:51.953105 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:29:51.953817 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:51.991596 18634 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.66:33009, user_credentials={real_user=slave}} blocked reactor thread for 52697.2us
I20240502 14:29:51.993662 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:52.023128 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:29:52.200227 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:29:52.283004 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:52.414156 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 14:29:52.416863 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:29:52.493994 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:52.519058 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:52.530062 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:52.536821 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:52.555121 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:52.563055 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:52.578014 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:52.585490 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:29:52.590162 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:52.597795 18769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.69:41119, user_credentials={real_user=slave}} blocked reactor thread for 52289.4us
I20240502 14:29:52.626842 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:52.627758 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:52.638971 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:29:52.642539 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:52.649431 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:52.663547 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:52.707856 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:52.720384 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:29:52.796535 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:29:52.850116 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:29:52.859696 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:29:52.867012 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:29:52.888592 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:29:52.924898 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:52.939060 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:52.970667 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:53.005502 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:29:53.044704 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:53.051254 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:53.065829 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:53.070703 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:53.094647 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:53.095108 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:53.136091 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:53.144990 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:53.158676 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:53.181815 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:53.192965 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:53.208281 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:29:53.212503 19032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 14:29:53.216332 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:29:53.360424 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:29:53.360908 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:29:53.369436 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:53.404909 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:53.406571 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:53.426689 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:53.429904 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:53.438997 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:53.468727 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:53.536868 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:29:53.547004 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:29:53.614076 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:29:53.649991 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:53.663997 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:53.675031 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:53.681984 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:53.689951 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:53.696617 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:53.698175 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:53.815428 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:53.871868 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:29:53.924209 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:29:53.944984 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:29:53.957556 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:29:53.973587 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:53.990442 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:54.007582 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:29:54.024866 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:54.037822 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:54.042049 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:54.129902 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:54.130759 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:29:54.161520 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:54.161994 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:54.179819 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:29:54.179795 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:29:54.222960 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:54.247223 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:54.260756 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:54.264892 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:54.270364 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:54.282409 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:54.475824 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:54.492206 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:54.495386 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:54.507478 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:54.508769 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:54.514168 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:54.604802 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:29:54.622118 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:29:54.647437 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:54.734941 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:29:54.769623 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:54.774916 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:54.812001 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:54.817559 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:54.822221 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:54.836475 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:54.880545 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:54.894318 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 14:29:54.996770 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:55.007577 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:55.046223 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:55.046734 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:55.067224 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:55.068518 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:55.075835 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:29:55.106177 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:55.163538 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:55.256258 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:29:55.277290 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:29:55.315721 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:29:55.318516 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:29:55.360491 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:55.377604 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:55.378223 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:55.403064 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:55.408159 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:29:55.410805 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:55.426087 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:55.440146 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:55.463094 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:29:55.496376 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:55.509510 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:29:55.531788 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:29:55.556325 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:29:55.564322 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:55.565279 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:55.572607 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:55.603184 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:55.624485 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:29:55.638126 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:29:55.670295 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:55.732950 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:55.754710 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:29:55.808216 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:55.825822 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:55.830207 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:55.837447 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:55.843290 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:55.867139 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:55.939677 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:29:55.968513 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:29:56.103521 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:56.119241 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:56.134169 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:56.150179 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:29:56.150996 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:56.158610 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:56.185534 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:56.192051 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:56.255857 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:29:56.309429 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:56.314599 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:29:56.325520 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:56.334013 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:56.343991 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:29:56.354346 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:56.363509 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:29:56.379971 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:56.385408 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:56.392967 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:56.445016 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:29:56.510206 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:29:56.582454 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:56.634763 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:29:56.663014 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:56.663722 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:56.675108 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:56.678292 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:56.682053 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:56.711150 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:56.736150 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 14:29:56.767572 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:29:56.776206 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:56.817523 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:29:56.865698 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:29:56.883819 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:29:56.928623 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:56.931610 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:56.972436 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:56.979849 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:57.002594 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:57.012712 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:57.017874 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:57.080806 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 82 similar messages]
I20240502 14:29:57.249681 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:57.264945 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:57.292340 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:57.294153 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:57.348017 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:29:57.356253 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:57.365171 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:57.381940 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:29:57.397835 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:57.409231 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:57.429561 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:57.450940 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:57.462409 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 14:29:57.467782 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:57.468995 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:57.527206 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:57.595933 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:57.696657 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:29:57.696990 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:29:57.714357 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:29:57.738138 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:29:57.759382 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:57.768993 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:29:57.778224 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:57.795481 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:57.812520 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:57.816890 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:57.847872 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:57.867126 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:29:57.888962 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:57.920886 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:57.941509 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:57.966387 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:57.970845 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:57.976002 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:29:57.986200 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:29:57.989464 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:58.004573 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:29:58.088603 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:58.100540 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:29:58.109421 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:29:58.166767 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:58.171129 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:29:58.238420 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:29:58.250890 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:29:58.293022 19032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 14:29:58.362089 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:29:58.416379 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:29:58.504478 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:58.513125 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:58.520423 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:58.549396 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:58.553205 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:58.570538 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:58.577543 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:58.591856 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:58.600054 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:58.607462 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:58.615101 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:58.656635 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:58.659474 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:29:58.708451 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:29:58.772049 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:29:58.807337 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:58.850035 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:29:58.923152 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:29:58.981266 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:29:58.988710 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:29:59.009064 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:29:59.027640 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:59.034691 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:59.035284 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:59.056921 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:59.077970 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:59.081600 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:59.081581 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:59.115291 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:59.118597 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:59.126622 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:59.127029 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:29:59.198652 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:29:59.203958 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:59.204834 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:59.218183 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:29:59.249011 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:29:59.325421 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:29:59.338132 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:29:59.472852 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:29:59.481272 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:59.482326 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:59.488348 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:59.488575 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:29:59.489001 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:59.642359 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:29:59.736320 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:59.753654 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:29:59.754675 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:29:59.755666 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:29:59.763959 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:29:59.763981 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:59.775533 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:29:59.802134 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:29:59.859584 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:30:00.025702 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:00.025913 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:00.047183 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:00.069046 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:00.101266 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:00.119520 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:30:00.126463 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:00.154726 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:30:00.194895 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 14:30:00.196359 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:30:00.232154 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:00.254791 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:00.268252 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:00.275506 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:30:00.281508 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:00.304082 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:00.306375 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:00.336117 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:30:00.351876 18900 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.69:41119, user_credentials={real_user=slave}} blocked reactor thread for 77809.9us
I20240502 14:30:00.359050 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:00.373778 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:30:00.379932 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:30:00.390430 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:00.482115 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:30:00.498965 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:30:00.531183 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:30:00.533715 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:30:00.584822 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:30:00.593833 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:00.596239 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:00.598809 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:00.606158 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:00.610582 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:00.611709 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:00.708662 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:30:00.819953 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:30:00.820955 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:00.867178 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:00.873843 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:00.879032 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:00.883391 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:00.904481 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:00.918715 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:00.918805 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:30:00.957612 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:01.082235 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:01.085875 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:01.127382 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:01.127892 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:01.162775 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:01.175194 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:01.217077 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:30:01.234767 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:30:01.263478 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:01.337599 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:01.338119 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:01.360817 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:01.360831 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:30:01.361126 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:01.363505 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:01.368978 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:01.386083 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:01.542800 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:30:01.551807 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:30:01.565873 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:30:01.618294 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:01.623353 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:01.631198 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:30:01.661135 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:01.671779 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:30:01.673454 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:01.687026 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:01.707140 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:01.760690 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240502 14:30:01.799300 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:30:01.800925 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:01.830416 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:30:01.898624 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:01.909551 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:30:01.946027 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:01.958966 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:01.959133 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:30:01.993705 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:01.994081 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:02.012380 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:02.052307 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:02.071592 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:02.109858 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:02.125600 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:02.125823 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:02.149009 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:02.179188 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:02.184697 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 14:30:02.276518 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:30:02.321266 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:02.485877 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 14:30:02.488909 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:02.491029 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:02.491537 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:02.501029 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:02.501621 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:02.501897 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:02.512921 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:02.538671 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:30:02.542274 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:02.561683 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:02.573828 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:02.604317 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:02.608686 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:02.636636 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:30:02.636768 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:30:02.639477 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:30:02.647015 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:02.675345 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:30:02.682152 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:30:02.701597 20562 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.67:45409, user_credentials={real_user=slave}} blocked reactor thread for 111581us
W20240502 14:30:02.715327 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:30:02.788071 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:02.878476 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:30:02.918774 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:30:02.935600 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:30:02.978514 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:30:03.006037 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:03.132763 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:03.180855 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:30:03.181555 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:03.190490 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:03.198165 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:03.209306 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:03.210651 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:03.214740 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:03.234555 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:30:03.264876 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:30:03.287434 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:30:03.307164 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:03.315394 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:03.343150 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:03.344215 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:03.363863 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:03.383252 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:03.416245 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:03.514900 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 14:30:03.539294 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:30:03.600767 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:03.635845 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:03.637305 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:03.642427 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:30:03.649025 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:03.678030 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:03.679144 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:03.680665 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:03.710983 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:30:03.826107 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:03.842203 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:03.843125 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:03.852344 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:03.859095 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:03.865667 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:03.872563 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:03.966091 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:30:03.972896 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:30:03.982396 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:30:04.001567 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:30:04.086886 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:30:04.092672 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:04.104192 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:04.125097 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:04.130276 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:04.130724 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:04.131139 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:04.156497 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:04.225934 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:30:04.310308 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:30:04.370393 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:04.378731 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:04.390558 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:04.415647 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:04.421898 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:04.421969 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:04.422834 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:30:04.436403 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:04.647245 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:30:04.666325 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:04.726279 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:04.764724 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:04.770273 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:04.779088 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:30:04.783928 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:04.785392 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:04.788069 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:04.791239 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:04.806942 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:04.851181 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:04.887706 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:30:04.888517 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:30:04.955591 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:04.982900 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:04.988684 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:04.995714 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:05.047900 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:30:05.057898 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:05.112828 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:30:05.158697 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:30:05.167094 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:05.187239 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:05.196411 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:05.200170 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:05.201747 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:05.206189 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:05.210511 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:30:05.252569 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:05.262208 20562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:30:05.365036 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:30:05.371686 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:30:05.378217 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:30:05.477773 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:05.477766 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:05.493124 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:30:05.503998 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:30:05.513707 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:05.528264 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:05.546255 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:05.550570 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:05.583668 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:05.660020 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:30:05.699039 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:05.717913 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:05.725044 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:05.729583 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:05.739847 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:30:05.745836 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:05.775653 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:05.780485 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:05.877233 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:30:05.902693 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:30:05.955452 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:05.965498 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:05.966610 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:05.978196 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:05.981757 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:06.002277 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:30:06.020875 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:06.032698 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:06.099356 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:30:06.178064 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:30:06.197440 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:06.209378 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:06.229995 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:06.235668 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:06.246241 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:06.264216 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:06.325582 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:06.332243 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:30:06.336395 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:30:06.428926 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:30:06.477320 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:30:06.501194 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:30:06.544829 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:06.559092 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:06.603624 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:06.630726 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:30:06.645915 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:06.647898 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:06.678419 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:06.684518 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:06.703366 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:06.713222 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:06.771448 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:06.775261 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 14:30:06.787658 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:30:06.803932 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:06.807216 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:06.826988 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:06.870831 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:06.951761 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:30:07.021145 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:30:07.022655 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:07.038045 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:30:07.063889 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:07.083304 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:07.109046 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:07.111312 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:07.114140 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:07.160151 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:07.238170 18637 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.68:41463, user_credentials={real_user=slave}} blocked reactor thread for 64402.1us
W20240502 14:30:07.268662 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:30:07.296640 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 14:30:07.299934 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:30:07.328639 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:07.341979 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:07.358814 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:07.361274 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:30:07.364046 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:07.367054 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:07.390774 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:07.430227 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:07.585446 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:07.594584 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 14:30:07.678764 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:07.680228 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:30:07.704237 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:07.716934 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:07.717530 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:07.724581 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:30:07.732870 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:07.751211 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:07.818897 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:30:07.826566 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:30:07.834596 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:30:07.902824 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:30:07.928385 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:30:07.966776 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:08.005067 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:30:08.022078 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:30:08.059655 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:08.066578 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:08.074601 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:08.090422 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:08.095921 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:08.100592 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:08.124703 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:08.205027 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:08.210209 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:08.213969 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:08.224264 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:08.232237 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:30:08.253783 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:30:08.259822 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:30:08.267141 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:08.278555 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:08.323139 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:30:08.362668 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:30:08.477090 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:08.499758 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:08.519368 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:30:08.525523 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:08.546432 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:30:08.549104 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:08.552876 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:08.565153 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:08.604899 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:08.645838 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 14:30:08.673400 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:08.732993 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:30:08.797030 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:30:08.809134 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:08.816082 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:08.876721 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:08.888981 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:08.894623 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:08.895195 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:30:08.905647 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:08.923775 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:30:08.971316 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:08.973384 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:30:08.981879 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:30:09.067699 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:09.068975 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:09.096451 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:09.103194 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:09.106978 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:09.110356 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:09.119205 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:30:09.172036 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:09.258770 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:30:09.292289 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:09.313084 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:09.316017 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:09.328428 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:09.340420 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:09.345567 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:09.561132 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:09.563230 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:09.568219 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:09.586916 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:09.587355 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:09.595067 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:30:09.597788 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:09.609534 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:09.668376 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:09.677728 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:30:09.807328 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:30:09.901731 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:09.920117 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:09.937223 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:09.949712 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:30:09.959236 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:09.973064 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:09.974026 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:10.027407 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:10.070086 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:10.097105 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:10.136268 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:10.152223 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:10.188241 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:10.243796 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:10.251149 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:10.253324 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:30:10.266577 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:30:10.273442 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 14:30:10.274435 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:30:10.431618 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:10.433632 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:10.459414 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:30:10.463346 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:30:10.464148 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:30:10.478366 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:10.483685 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:10.493719 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:10.515726 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:10.573478 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:10.574110 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:30:10.587158 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:30:10.661566 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:30:10.693957 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:10.724783 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:10.740051 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:10.740700 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:10.741083 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:10.741479 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:10.743849 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:10.761375 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:30:10.776082 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:30:10.802461 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:30:10.803460 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:30:10.818889 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:30:10.976560 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:30:11.020298 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:11.030018 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:11.040411 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:30:11.045872 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:11.067720 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:11.086557 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:11.097472 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:11.099479 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:11.108649 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:11.319365 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:30:11.331315 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:30:11.345801 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:11.362746 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:11.369297 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:11.380896 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:11.394244 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:11.421036 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:11.423080 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:30:11.456069 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:30:11.524353 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:11.546389 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:30:11.561185 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:30:11.599072 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:30:11.599398 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:11.611330 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:11.617110 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:11.655088 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:11.687376 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:11.688238 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:11.689504 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:11.839406 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 90 similar messages]
I20240502 14:30:11.846814 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:11.854700 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:11.854851 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:11.857705 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:11.888406 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:11.916786 20562 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.71:39195, user_credentials={real_user=slave}} blocked reactor thread for 54615.5us
I20240502 14:30:11.925246 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:11.940259 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:11.978192 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:30:12.047384 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:30:12.050199 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:12.171721 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:12.177716 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:12.186556 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:30:12.188644 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:12.194415 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:12.230528 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:12.247252 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:12.299618 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:12.329983 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:12.335330 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:30:12.336803 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:12.337507 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 82 similar messages]
I20240502 14:30:12.365003 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:12.391085 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:12.393631 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:30:12.431304 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:12.471845 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:12.484349 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:12.505757 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:30:12.616811 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:12.628046 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:12.667603 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 14:30:12.668812 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:12.682859 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:12.686224 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:12.738365 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:12.759380 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:30:12.764835 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:12.804973 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:30:12.818190 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:30:12.867818 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:30:12.912642 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:12.928251 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:12.933526 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:12.933939 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:12.934880 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:30:12.937316 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:12.937422 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:12.976158 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:30:12.996629 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:12.996798 18769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.69:41119, user_credentials={real_user=slave}} blocked reactor thread for 49770.5us
W20240502 14:30:13.044377 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:30:13.055369 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:30:13.075601 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:30:13.125094 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:13.127142 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:30:13.160416 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:30:13.231280 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:13.237897 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:13.254637 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:13.273654 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:30:13.303884 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:13.305614 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:13.314822 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:13.335433 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:30:13.407042 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:13.421525 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:30:13.434818 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:13.440675 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:13.447605 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:13.464253 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:13.465693 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:30:13.476971 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:13.530158 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:30:13.548552 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:13.575937 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:13.819074 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:13.832437 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:13.847555 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:13.862275 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:13.865401 19032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 14:30:13.874651 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:13.883937 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:13.887642 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:13.887997 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:13.929201 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:13.951915 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:30:13.960350 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:13.968873 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:30:13.979437 20606 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:13.979524 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:14.008045 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:30:14.087266 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:30:14.094732 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:14.106849 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:14.134858 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:30:14.150993 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:30:14.209906 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:30:14.463908 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:30:14.497538 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:14.507480 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:14.510775 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:14.551847 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:14.557068 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:14.563414 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:14.621757 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:14.623935 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:30:14.624775 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:14.637475 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:14.637934 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:14.645535 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:14.681175 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:14.691857 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:14.693017 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:14.708531 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:30:14.917467 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:30:14.966013 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:30:14.989431 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:15.001946 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:15.033859 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:15.069051 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:15.074894 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:15.087148 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:15.103818 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:15.108052 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:15.123510 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:30:15.132541 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:15.180568 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:15.182461 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:15.187773 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:15.209467 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:15.262012 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:30:15.288954 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:15.372428 20562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:30:15.423929 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:30:15.430527 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:30:15.435932 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:30:15.479449 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:30:15.500449 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:15.502676 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:15.510113 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:30:15.522917 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:15.524935 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:15.533596 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:15.537487 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:15.558843 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:30:15.559129 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:30:15.578406 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:30:15.627409 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:30:15.633080 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:15.635136 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:30:15.649576 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:30:15.746620 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:30:15.826146 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:15.832067 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:15.839983 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:15.850416 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:15.854238 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:15.869751 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:15.870894 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:15.884426 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:30:16.010660 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:30:16.077286 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:16.094480 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:16.097317 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:16.145495 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:16.147692 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:16.162043 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:16.198450 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:16.302845 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:16.314746 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:16.336915 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:16.377290 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:30:16.382691 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:16.398574 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:16.419225 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:16.426546 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:16.473497 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:30:16.490370 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:30:16.565308 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:30:16.622364 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:16.640987 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:16.656814 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:16.659770 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:16.675886 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:16.677013 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:16.692672 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:30:16.693109 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:16.816226 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:30:16.828840 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:30:16.839742 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:30:16.842550 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:30:16.924360 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 90 similar messages]
I20240502 14:30:16.936893 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:16.949168 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:16.955197 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:16.960237 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:16.967715 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:17.008250 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:17.008601 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:17.133085 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:17.139407 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:17.148558 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:17.165906 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:17.167479 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:17.170733 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:30:17.189740 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:17.195848 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:17.241883 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:30:17.347448 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:30:17.374135 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 81 similar messages]
I20240502 14:30:17.425665 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:17.429414 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:17.448256 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:17.484761 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:17.497431 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:17.536221 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:30:17.540635 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:17.560942 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:17.609490 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:30:17.629935 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:30:17.642704 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:17.750938 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:17.763546 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:17.764889 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 14:30:17.765532 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:17.771138 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:17.781507 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:17.798046 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:17.854846 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:30:17.871877 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:30:17.924940 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:30:17.985245 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:18.019271 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:30:18.060360 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:18.063546 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:18.080348 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:18.114794 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:18.117754 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:18.123597 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:18.126083 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:18.143142 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:30:18.160202 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:18.160231 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:18.166311 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:18.166715 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:30:18.169528 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:30:18.173774 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:18.180053 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:30:18.188570 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:30:18.190811 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:30:18.209929 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:30:18.230688 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:30:18.257398 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:30:18.260469 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:18.278970 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:18.503286 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:18.528730 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:18.540233 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:18.548285 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:18.600613 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:18.602423 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:18.604326 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:18.636654 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:18.692656 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:30:18.777550 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:18.788020 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:18.805884 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:18.812059 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:18.817514 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:18.818707 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:18.885406 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:18.925441 19032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 14:30:18.977587 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:30:19.019111 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:30:19.055141 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:30:19.055423 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:30:19.089327 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:19.092057 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:19.097519 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:19.104496 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:19.106349 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:19.111701 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:30:19.130004 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:19.149722 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:19.168686 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:30:19.244079 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:30:19.304944 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:19.359774 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:19.400233 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:19.432351 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:19.449946 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:30:19.455749 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:19.461478 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:19.477384 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:19.477674 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:30:19.615820 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:19.661645 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:19.670843 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:19.681135 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:19.701764 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:19.702384 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:30:19.715526 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:19.720042 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:30:19.734961 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:19.881907 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:19.890077 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:19.900276 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:19.917317 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:19.923455 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:19.938062 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:19.953969 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:20.002810 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:30:20.160164 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:30:20.162744 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:20.199466 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:20.208678 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:20.224591 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:20.225615 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:20.228137 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:20.265846 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:20.271759 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:30:20.282361 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:30:20.283694 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:20.305126 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:20.333148 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:30:20.337117 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:20.360203 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:20.361275 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:30:20.364516 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:20.395560 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:30:20.396057 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:20.419116 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:30:20.422372 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:30:20.423293 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:30:20.434775 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:30:20.476636 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:20.559340 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:30:20.611630 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:20.624641 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:20.630286 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:30:20.644994 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:30:20.655891 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:20.661146 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:20.664511 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:20.667905 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:20.720402 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:20.727489 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:30:20.738513 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:30:20.771582 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:30:20.910984 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:20.934787 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:20.942575 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:20.947355 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:20.964740 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:20.974856 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:21.007516 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:21.010305 20562 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.71:39195, user_credentials={real_user=slave}} blocked reactor thread for 51473.4us
W20240502 14:30:21.201238 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:21.205408 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:30:21.299563 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:21.329794 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:21.338977 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:21.363677 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:21.364703 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:21.371371 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:21.374415 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:21.375396 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:21.394196 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:21.398660 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:30:21.414028 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:30:21.428553 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:21.432332 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:21.443805 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:21.466817 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:21.705685 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:21.726984 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:30:21.761845 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:21.777397 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:30:21.781366 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:30:21.787297 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:21.821453 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:30:21.850433 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:21.853302 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:21.871627 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:21.877382 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:21.914973 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:21.919394 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:30:21.938256 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 88 similar messages]
I20240502 14:30:21.940062 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:21.943852 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:30:21.960112 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:30:21.963225 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:21.994704 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:30:21.997256 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:22.017393 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:22.025036 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:22.046365 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:22.151831 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:22.312624 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:22.319274 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:22.346421 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:22.358068 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:22.380175 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:22.396787 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:22.401816 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:22.418673 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:30:22.433254 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 14:30:22.447947 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:30:22.522195 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:30:22.612474 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:22.616247 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:22.644297 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:30:22.666652 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:22.686368 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:22.713793 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:22.727293 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:30:22.782166 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:22.800720 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:22.810104 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 14:30:22.844333 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:22.853031 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:30:22.854943 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:22.873384 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:30:22.915714 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:22.920256 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:22.922698 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:22.938247 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:22.958698 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:30:22.967937 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:30:22.979445 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:30:22.981451 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:22.996553 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:30:23.006945 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:23.033475 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:30:23.048163 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:30:23.101459 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:30:23.168339 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:23.175190 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:30:23.177157 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:23.179661 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:23.180363 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:23.182602 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:23.190052 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:23.211865 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:30:23.264266 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:30:23.389017 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:30:23.478946 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:23.499167 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:23.520906 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:23.521986 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:23.551055 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:23.585937 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:23.591270 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:23.617523 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:23.633313 20964 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.5.107.71:39195, user_credentials={real_user=slave}} blocked reactor thread for 70539.8us
W20240502 14:30:23.636757 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:23.647691 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:23.649575 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:23.651860 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:23.679250 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:23.754783 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:23.854591 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:30:23.857133 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:30:23.870507 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:30:23.925839 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:23.933498 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:23.939429 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:23.948379 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:23.950963 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:23.951881 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:23.952802 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:23.963460 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 14:30:23.981416 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:30:24.125571 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:24.155153 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:30:24.182142 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:30:24.190649 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:24.196060 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:24.199318 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:24.203871 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:24.209414 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:24.240113 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:24.343425 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:30:24.370771 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:30:24.395584 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:30:24.398330 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:30:24.427971 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:30:24.468130 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:24.534358 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:24.559687 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:24.579991 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:24.642374 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:24.652654 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:24.659464 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:24.671409 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:24.673159 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:24.681577 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:24.709267 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:24.714975 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:24.773438 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:24.776417 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:24.791786 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:30:24.913865 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:24.990002 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:25.080729 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:25.097471 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:25.125002 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:25.148767 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:25.153967 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:25.172570 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:25.197578 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:30:25.199124 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:25.199257 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:30:25.223866 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:25.248647 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:25.283969 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:25.303020 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:30:25.312860 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:25.313455 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:25.331921 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:30:25.357580 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:30:25.441740 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:25.442230 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:30:25.462865 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:25.534765 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:30:25.539863 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:30:25.586509 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:30:25.592140 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:30:25.597590 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:30:25.603726 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:30:25.616271 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:30:25.619065 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:30:25.661410 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:30:25.692554 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:30:25.710423 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:30:25.838001 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:25.839215 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:25.849828 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:25.864333 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:25.887305 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:25.901654 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:25.921389 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:25.938153 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:25.938253 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:25.938314 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:25.992563 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:26.021116 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:26.030999 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:26.041512 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:26.199347 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:30:26.272194 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:30:26.294013 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:26.331105 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:30:26.337282 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:26.337639 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:26.338207 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:26.354849 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:26.358773 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:30:26.367516 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:26.378492 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:26.385432 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:26.434633 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:26.446885 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:26.476217 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:26.491974 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:26.492981 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:26.507892 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:26.605880 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:30:26.735378 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:26.737501 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:26.746526 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:26.749634 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:30:26.764891 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:30:26.792871 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:26.796597 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:30:26.803310 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:26.817497 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:26.823069 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:30:26.836313 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:26.884552 18769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.71:39195, user_credentials={real_user=slave}} blocked reactor thread for 87965.1us
W20240502 14:30:26.898605 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:30:26.947870 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 14:30:26.948906 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:26.964416 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:30:27.009227 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:27.009321 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:27.012753 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:27.025583 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:27.036391 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:27.036635 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:27.245311 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:30:27.318631 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:27.338032 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:27.342666 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:27.344342 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:27.345525 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:27.350421 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:27.395296 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:27.400434 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:27.430542 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:27.465772 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:27.473233 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:30:27.490229 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:27.512096 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:27.524135 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 14:30:27.525326 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:30:27.548008 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:27.548913 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:27.677150 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:30:27.756182 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:27.760272 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:27.769017 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:27.770697 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:27.780869 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:27.793299 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:27.810225 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:27.830765 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:30:27.865891 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:27.987991 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 14:30:28.006516 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:30:28.018280 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:30:28.020479 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:30:28.023535 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:30:28.027166 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:30:28.062530 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:28.077991 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:30:28.086095 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:30:28.087847 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:30:28.134119 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:28.140085 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:30:28.159200 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:28.162075 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:28.198256 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:30:28.218712 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:30:28.222990 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:28.233623 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:28.245131 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:28.248982 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:28.261569 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:28.269418 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:28.293850 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:28.294482 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:30:28.301515 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:30:28.310506 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:28.311087 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:28.402261 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:28.577533 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:28.586187 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:28.595916 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:28.611227 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:28.614567 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:28.622288 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:30:28.626199 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:28.701131 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:30:28.715397 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:30:28.739638 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:30:28.785887 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:28.856809 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:28.863162 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:28.873633 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:28.882236 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:28.887041 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:28.902287 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:28.913763 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:29.099298 19032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 14:30:29.113165 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:30:29.129577 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:30:29.149603 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:29.150826 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:30:29.163842 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:29.177742 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:29.181164 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:29.198940 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:29.221655 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:29.222306 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:29.252775 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:30:29.353789 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:29.363545 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:30:29.367245 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:29.400483 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:29.435534 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:29.436872 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:29.458977 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:29.505955 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:29.560992 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:30:29.641553 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:29.649073 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:29.693432 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:30:29.695655 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:29.696728 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:30:29.699049 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:29.706588 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:29.735525 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:29.751988 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:29.758271 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:30:29.886981 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:29.887501 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:29.897475 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:29.899516 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:29.903291 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:29.917126 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:30:29.964922 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:30.061774 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:30.142050 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:30.173621 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:30.192706 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:30.194622 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:30.198119 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:30.210268 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:30:30.222083 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:30.224987 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:30.276577 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:30:30.341576 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:30:30.348008 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:30:30.448128 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:30.455185 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:30:30.498828 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:30:30.523862 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:30:30.532989 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:30.549757 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:30.551251 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:30.564275 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:30:30.573714 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:30.578449 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:30.581405 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:30.594759 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:30:30.608124 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:30:30.619318 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:30.632241 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:30.634438 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:30.636955 20562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:30:30.645123 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:30:30.673620 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:30.695472 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:30.710557 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:30:30.720309 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:30.724598 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:30.812582 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:30:30.926868 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:30:30.944965 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:31.031530 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:31.032296 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:31.037911 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:31.040014 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:31.047216 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:31.048316 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:31.095522 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:30:31.123023 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:30:31.150961 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:31.166929 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:30:31.262885 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:30:31.326577 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:31.360529 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:31.383474 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:31.385599 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:31.406738 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:31.408035 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:31.409411 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:31.439230 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:30:31.442377 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:30:31.469930 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:31.486028 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:31.507841 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:31.511947 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:31.527873 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:31.552587 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:31.676631 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:31.681898 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:30:31.750156 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:31.755136 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:31.767417 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:31.770735 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:31.785985 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:31.800789 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:31.801879 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:30:31.854102 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:31.875371 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:30:31.972398 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 14:30:32.004033 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:30:32.017683 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:32.035156 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:32.035813 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:32.037079 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:32.039597 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:32.063767 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:32.080036 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:32.113792 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:30:32.318202 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:32.339150 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:32.349366 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:32.357307 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:32.380560 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:32.385926 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:30:32.417361 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:32.427536 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:30:32.467068 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:32.515069 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:32.527788 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 14:30:32.535075 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:32.548197 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:30:32.548381 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:30:32.559700 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:32.575218 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:32.607266 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:32.624001 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:32.628262 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:32.796320 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:30:32.813102 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:30:32.898329 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:32.907694 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:32.921860 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:32.937109 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:30:32.945319 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:32.946292 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:32.954360 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:32.974329 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:32.993873 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:33.003177 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:33.036196 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:30:33.040194 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:33.082942 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:33.090512 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:33.099562 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 14:30:33.100615 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:30:33.110061 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:30:33.125319 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:30:33.126616 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:30:33.129912 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:30:33.149232 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:33.172749 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:33.226622 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:30:33.246028 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:30:33.340570 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:30:33.378283 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:30:33.467519 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:30:33.469143 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:33.555737 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:30:33.585700 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:33.616396 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:33.620414 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:33.641541 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:33.656293 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:33.668953 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:33.702965 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:33.705003 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:30:33.788949 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:33.808957 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:30:33.829278 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:33.842427 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:30:33.846422 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:33.849891 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:30:33.854851 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:33.857451 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:33.859021 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:33.974650 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:33.977902 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:30:34.090148 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:30:34.110792 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:34.148285 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 14:30:34.170924 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:30:34.176508 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:34.184898 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:34.185109 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:34.198577 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:34.203369 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:34.219687 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:34.220072 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:34.264279 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:34.328752 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:30:34.354537 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:34.359297 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:34.407428 20561 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.72:35871, user_credentials={real_user=slave}} blocked reactor thread for 45782.5us
I20240502 14:30:34.407928 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:34.413738 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:34.427338 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:30:34.532966 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:34.616515 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:34.628934 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:34.651472 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:34.655913 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:34.678299 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:34.713882 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:34.714391 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:34.760738 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:30:34.801959 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:30:34.852726 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:30:34.893323 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:34.901682 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:34.920393 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:34.927018 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:34.940971 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:34.950064 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:34.967366 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:34.981541 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:30:35.112145 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:30:35.114374 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:35.224478 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:35.231952 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:30:35.258497 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:35.261463 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:35.265036 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:30:35.281522 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:35.333376 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:35.346380 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:35.351809 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:35.377321 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:35.379683 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:35.419752 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:35.462064 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:35.487658 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:30:35.488059 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:35.495113 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:35.510937 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:35.511245 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:30:35.519549 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:30:35.658360 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:30:35.660226 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 14:30:35.661597 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:30:35.727561 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:30:35.728535 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:35.731813 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:35.737011 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:35.741518 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:35.742722 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:35.758653 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:30:35.774987 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:30:35.788149 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:35.811036 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:30:35.819545 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:30:35.822679 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:30:35.918777 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:30:35.937323 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:36.032351 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:36.052760 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:36.073271 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:36.105942 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:36.110563 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:30:36.114220 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:36.123129 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:36.128684 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:36.158509 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:36.187243 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:36.287515 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:30:36.290132 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:36.336129 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:36.336514 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:30:36.344482 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:36.375638 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:36.475709 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:36.545449 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:36.549901 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:36.563390 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:30:36.564738 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:36.572454 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:30:36.602852 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:36.604063 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:36.604502 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:36.640309 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:36.726629 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:30:36.817425 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:36.835886 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:36.840999 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:36.841933 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:36.842795 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:36.860059 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:30:36.862548 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:36.889046 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:36.988957 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 14:30:36.989923 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:30:37.015548 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:30:37.024292 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:37.120498 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:37.130403 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:30:37.133215 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:37.150730 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:37.155478 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:37.178400 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:30:37.192060 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:37.195346 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:37.259084 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:37.259604 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:37.262586 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:30:37.267527 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:37.313890 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:37.327023 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:37.358551 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:37.392017 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:37.470569 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:30:37.475620 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:30:37.562800 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240502 14:30:37.606316 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:30:37.624483 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:37.635936 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:37.662262 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:37.663750 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:30:37.677656 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:37.691356 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:37.703101 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:37.712363 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:37.717559 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:37.729436 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:37.765595 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:37.789290 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:37.846525 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:37.881713 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:37.906306 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:37.953408 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:30:37.958200 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:30:37.970252 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:30:37.971470 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:30:38.211954 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:38.253391 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:30:38.305950 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:30:38.332353 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 14:30:38.339414 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:38.339808 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:30:38.340210 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:30:38.360666 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:38.371613 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:30:38.374632 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:38.376192 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:38.385133 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:38.391402 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:30:38.399169 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:30:38.411247 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:38.426702 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:38.447032 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:30:38.484308 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:30:38.543716 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:38.547156 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:38.567718 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:38.583866 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:38.595721 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:38.632102 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:30:38.634117 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:38.641490 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:38.648828 18637 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.67:45409, user_credentials={real_user=slave}} blocked reactor thread for 99494.4us
W20240502 14:30:38.710943 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:30:38.814318 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:30:38.823786 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:30:38.903874 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:38.920974 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:38.923914 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:38.944582 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:38.954281 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:38.973659 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:30:38.973798 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:39.003523 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:39.134295 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:39.143850 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:39.157675 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:39.177338 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:39.215997 19032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 14:30:39.218010 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:39.219365 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:39.283653 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:39.349257 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:30:39.422240 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:39.424573 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:30:39.436316 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:39.440768 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:39.442013 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:39.456914 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:30:39.460466 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:39.478224 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:39.482935 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:39.669447 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:30:39.722785 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:39.725256 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:39.728001 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:30:39.728417 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:39.729003 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:30:39.731658 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:39.732206 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:39.753772 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:39.757094 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:30:39.783900 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:39.867892 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:30:39.878741 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:30:39.994642 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:40.004519 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:30:40.044124 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:40.051494 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:40.054098 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:40.066726 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:40.072355 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:40.075158 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:40.102344 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:30:40.215698 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:40.228545 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:40.248407 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:40.261943 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:40.267983 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:30:40.271543 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:40.291877 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:40.325727 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:30:40.350991 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:40.395797 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:30:40.451483 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:30:40.507174 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:40.522504 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:40.541831 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:40.563863 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:40.566358 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:40.571254 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:40.635288 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:40.732533 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:30:40.814604 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:40.819222 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:30:40.826733 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:30:40.863477 20562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:30:40.867839 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:30:40.873107 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:30:40.887871 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:40.915396 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:30:40.916525 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:30:40.946949 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:40.976493 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:40.995973 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:40.997051 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:41.043514 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:41.060326 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:41.073437 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:41.084460 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:41.103403 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:30:41.110885 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:30:41.111789 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:30:41.130584 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:41.146005 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:41.193138 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:41.210011 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:41.284320 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:41.301324 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:30:41.346709 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:41.357007 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:41.368741 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:41.369832 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:41.377698 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:41.378820 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:30:41.414355 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:41.600723 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:41.604696 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:41.612931 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:41.615331 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:41.616505 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:41.623176 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:41.630405 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:30:41.631897 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:41.651558 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:30:41.802860 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:30:41.887491 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:30:41.930922 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:41.947520 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:41.952981 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:41.956633 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:41.994225 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 88 similar messages]
I20240502 14:30:41.999385 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:42.013068 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:42.033241 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:42.058329 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:42.063967 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:42.092895 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:42.137020 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:30:42.142782 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:30:42.161958 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:30:42.198133 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:30:42.216575 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:30:42.226389 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:42.227142 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:42.227836 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:42.287473 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:42.389885 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:30:42.427909 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:30:42.481314 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:42.491468 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:42.584599 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:42.592420 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:42.592864 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:42.604344 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 81 similar messages]
I20240502 14:30:42.607873 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:42.608569 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:30:42.618297 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:42.631961 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:42.677788 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:42.710605 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:42.785343 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:42.807550 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:42.814898 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:42.818642 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:30:42.843956 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:42.859277 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:30:42.946871 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:30:42.971508 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:30:42.982539 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:43.021927 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:43.183135 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:43.187886 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:43.193534 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:30:43.198912 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:43.210692 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:30:43.211324 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:43.213217 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:43.219133 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:43.237344 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:30:43.316071 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:30:43.326097 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:30:43.326881 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:30:43.461309 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:43.471256 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:43.485445 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:43.494586 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:30:43.503170 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:43.520200 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:43.523833 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:43.531863 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 14:30:43.534618 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:30:43.558952 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:43.591388 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:43.728672 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:43.740355 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:30:43.743605 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:43.765915 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:30:43.779174 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:43.785778 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:43.809834 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:30:43.812579 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:43.824018 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:43.924387 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:30:43.967164 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:43.976823 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:30:43.978339 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:43.982007 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:44.005589 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:44.030684 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:44.034854 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:44.040874 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:44.074131 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:44.136428 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:30:44.151391 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:30:44.174820 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:30:44.209285 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:44.225637 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:44.248270 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:44.248646 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:44.250795 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:44.254125 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 14:30:44.286226 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:44.381664 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:30:44.429334 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:44.483464 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:44.485318 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:44.485934 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:44.497323 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:30:44.502104 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:44.505726 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:44.513499 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:44.548178 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:44.563051 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:30:44.652032 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:30:44.705862 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:30:44.766619 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:30:44.814266 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:30:44.832728 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:30:44.863222 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:44.866155 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:44.881479 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:44.890102 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:44.893196 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:44.893831 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:44.937216 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:45.009881 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:45.010660 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:45.041785 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:45.042500 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:45.057703 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:45.062171 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:45.101702 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:45.102747 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:30:45.210533 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:30:45.243150 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:45.245491 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:45.257233 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:45.261598 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:45.283792 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:45.312219 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:45.312937 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:30:45.460973 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:30:45.497561 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:30:45.517696 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:45.553748 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:45.566316 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:45.594415 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:45.597088 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:45.608955 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:30:45.630156 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:45.644341 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:30:45.668087 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:45.708534 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:30:45.711174 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:45.739959 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:30:45.836218 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:30:45.845793 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:45.862632 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:45.889509 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:45.893936 20562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:30:45.895339 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:30:45.911266 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:45.912528 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:45.919893 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:45.961813 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:30:45.980042 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:30:46.083653 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:46.092368 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:30:46.097474 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:46.102028 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:46.107560 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:46.108386 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:46.126015 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:46.126757 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:46.163076 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:46.271608 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:30:46.394898 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:46.405468 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:46.430809 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:46.436648 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:46.443604 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:46.473729 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:46.561522 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:30:46.592833 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:30:46.594633 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:30:46.636770 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:46.718927 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:46.722656 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:30:46.740366 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:30:46.765857 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:46.769435 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:46.788856 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:46.810859 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:46.813797 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:30:46.815469 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:46.860034 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:46.923050 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:46.945719 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:46.961447 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:30:46.984201 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:30:47.005745 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:47.016748 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:47.034667 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:47.034669 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:47.041678 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 14:30:47.052461 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:30:47.086185 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:30:47.102157 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:47.228809 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:47.239389 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:30:47.251780 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:47.268110 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:47.275540 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:47.304842 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:47.332170 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:30:47.333492 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:47.361173 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:47.429370 20961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.66:33009, user_credentials={real_user=slave}} blocked reactor thread for 45621.3us
W20240502 14:30:47.448706 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:30:47.498512 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:47.528759 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:47.540213 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:47.543896 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:47.546584 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:47.548998 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:47.577368 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:47.591279 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:30:47.619462 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 14:30:47.723801 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:47.737752 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:30:47.788694 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:30:47.798959 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:30:47.864136 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:47.869607 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:47.891579 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:47.896036 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:47.925060 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:47.927489 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:47.931950 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:30:48.026227 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:30:48.052029 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:30:48.057729 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:48.226686 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:48.228842 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:48.255271 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:48.265460 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:48.280998 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:30:48.283680 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:48.300905 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:48.342630 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:30:48.344218 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:48.345338 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:48.366607 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:48.387143 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:48.393951 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:48.394526 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:48.405596 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:48.483270 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:30:48.499049 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:30:48.518834 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:30:48.565589 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:30:48.566838 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:48.623466 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 14:30:48.709635 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:48.710759 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:48.714109 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:48.718851 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:48.726436 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:48.731904 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:48.739216 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:48.749327 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:30:48.844403 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:30:49.018395 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:49.035454 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:49.060936 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:49.063933 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:30:49.070674 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:49.104785 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:49.116114 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:49.124457 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:30:49.139725 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:49.145368 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:30:49.148305 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:49.174865 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:49.193105 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:30:49.202270 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:49.203951 18636 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.72:35871, user_credentials={real_user=slave}} blocked reactor thread for 46354.4us
W20240502 14:30:49.225286 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:30:49.225456 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:30:49.229251 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:49.243441 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:30:49.253259 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:49.274650 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:49.293525 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:49.346534 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:30:49.382175 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:30:49.387926 19032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 14:30:49.394599 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:30:49.490409 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:49.492085 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:49.508728 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:49.509860 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:49.513480 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:49.527807 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:49.644266 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:49.647179 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:30:49.737957 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:30:49.794023 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:49.816779 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:49.819677 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:49.826792 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:49.842235 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:49.865679 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:49.881062 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:49.938349 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:30:49.961606 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:30:49.990377 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:30:50.037922 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:50.118417 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:30:50.150702 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:50.160648 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:50.162591 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:50.175367 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:50.186838 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:50.190778 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:30:50.202078 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:50.225257 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:50.226079 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:30:50.233454 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:50.249986 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:30:50.258097 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:50.277372 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:50.293324 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:50.308071 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:50.309197 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:50.475441 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:30:50.559932 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:30:50.569000 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:30:50.628167 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:50.637435 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:50.646898 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:50.648200 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:50.701606 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:50.704326 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:50.740159 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:30:50.766496 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:50.786083 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:50.787271 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:50.864745 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:30:50.875181 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:50.877455 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:50.895828 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:50.896834 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:50.911916 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:50.928504 20562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:30:50.941612 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:30:50.995713 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:30:51.089406 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:51.204465 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:30:51.228384 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:30:51.390988 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:30:51.404997 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:51.428437 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:51.458873 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:51.477263 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:51.480814 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:51.493860 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:51.507510 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:51.511762 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:51.527280 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:51.540702 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:51.556082 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:51.576694 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:51.587379 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:51.627475 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:51.633780 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:30:51.683951 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:30:51.688390 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:30:51.704243 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:30:51.708868 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:30:51.714272 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:30:51.863446 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:30:51.870405 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:51.894505 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:51.897548 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:51.920480 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:51.923908 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:30:51.940550 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:51.966149 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:51.975014 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:52.003412 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:30:52.059803 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:52.089090 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:52.099254 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 90 similar messages]
I20240502 14:30:52.106870 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:52.142385 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:52.153717 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:52.157876 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:52.208271 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:52.287324 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:30:52.347681 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:30:52.365334 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:30:52.378468 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:52.385932 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:52.388545 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:52.394861 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:52.423220 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:30:52.424747 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:52.425689 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:52.479452 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:30:52.523056 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:52.577987 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:30:52.595291 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:30:52.644513 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:52.645699 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:52.650920 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:52.654346 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:52.662776 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:52.665743 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:52.715448 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:52.723541 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 14:30:52.883482 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:30:52.920696 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:52.926054 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:52.928205 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:52.961700 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:30:52.974735 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:52.977188 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:53.000619 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:53.048272 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:30:53.052312 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:53.113711 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:53.115552 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:53.146368 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:53.154042 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:53.161516 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:53.191267 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:30:53.194904 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:53.200119 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:30:53.208575 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:53.233515 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:30:53.362197 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:30:53.372529 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:53.379786 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:30:53.380864 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:53.385550 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:53.388056 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:53.393954 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:53.409909 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:53.503360 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:30:53.543422 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:30:53.636076 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:53.650580 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:53.661551 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 14:30:53.672816 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:53.721961 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:53.724246 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:53.739312 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:53.755803 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:53.777608 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:53.779712 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:53.781808 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:53.820343 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:30:53.828575 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:53.840548 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:53.845535 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:53.881373 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:30:53.896611 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:54.044937 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:30:54.076071 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:30:54.077520 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:30:54.081149 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:54.083463 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:54.089067 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:54.092777 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:54.096998 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:54.102443 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:54.212750 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:54.226246 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:30:54.237282 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240502 14:30:54.238672 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:54.265902 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:30:54.295959 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:30:54.328471 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:54.338066 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:54.345121 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:54.350608 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:54.362524 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:54.367514 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:54.538996 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:30:54.571578 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:30:54.604636 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:54.632763 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:54.638726 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:54.665215 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:54.669293 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:30:54.677610 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:30:54.681859 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:54.692410 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:54.770541 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:54.773674 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:54.787714 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:54.794934 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 14:30:54.811925 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:30:54.816418 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:54.816622 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:54.825439 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:54.840533 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:54.859141 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:54.927803 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:30:54.943661 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:30:55.072012 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:55.087092 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:55.115533 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:55.133276 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:30:55.137671 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:55.163226 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:55.184738 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:55.186293 18637 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.69:41119, user_credentials={real_user=slave}} blocked reactor thread for 58759.7us
I20240502 14:30:55.201419 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:55.223647 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:30:55.253440 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:55.257757 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:55.281687 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:55.298611 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:55.309276 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:30:55.329627 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:55.338534 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:55.346809 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:55.502207 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:30:55.689620 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:30:55.757704 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:30:55.773737 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:55.795868 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:30:55.802388 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:30:55.844237 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:30:55.916352 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:55.930639 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:55.933583 20562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:30:55.934689 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:30:55.946192 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:55.957171 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:30:55.965003 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:55.984035 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:55.984712 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:55.985834 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:56.027073 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:56.047179 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:56.052419 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:56.052453 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:56.052925 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:30:56.079442 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:30:56.105964 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:56.106611 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:56.251655 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:30:56.333315 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:30:56.333729 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:56.377961 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:56.416786 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:56.458091 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:56.470237 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:56.476831 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:56.479151 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:56.481509 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:30:56.488842 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:56.497167 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:56.523123 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:56.550351 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:30:56.594451 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:56.599704 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:56.605465 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:56.655849 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:56.668833 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:30:56.688560 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:30:56.786361 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:56.864571 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:56.896154 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:56.899044 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:56.919116 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:30:56.923007 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:56.948271 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:56.951236 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:56.969499 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:30:56.981580 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:57.095391 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:30:57.095620 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:30:57.213023 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:57.224172 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:57.224607 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:57.226549 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 90 similar messages]
I20240502 14:30:57.233883 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:57.245455 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:57.253458 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:57.259719 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:57.269980 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:30:57.308413 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:30:57.337710 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:30:57.348309 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:30:57.454304 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:30:57.461956 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:30:57.467622 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:57.519405 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:57.560351 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:57.573791 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:57.579317 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:57.581679 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:57.583420 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:57.598063 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:57.602768 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:57.630808 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:57.633594 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:57.657594 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:30:57.658131 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:57.689648 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:57.703482 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:57.798923 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 14:30:57.894270 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:57.985141 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:58.002050 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:58.009255 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:58.012820 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:58.032184 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:58.033012 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:58.098199 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:30:58.115604 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:30:58.208042 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:30:58.231196 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:30:58.255941 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:58.279453 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:58.283375 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:30:58.300571 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:58.303812 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:30:58.309563 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:58.324213 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:30:58.338169 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:58.339757 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:58.372458 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:30:58.418495 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:30:58.457051 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:58.471307 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:58.473239 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:58.474045 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:58.482923 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:58.489091 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:58.497062 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:58.509591 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:30:58.518333 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:30:58.581267 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:30:58.641453 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:30:58.652935 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:30:58.663511 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 57 similar messages]
I20240502 14:30:58.712749 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:58.714684 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:58.725425 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:58.733217 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:58.733479 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:58.741497 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:30:58.898736 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:58.900436 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:58.951974 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:30:58.996275 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:59.017627 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:59.021077 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:59.025761 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:59.046833 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:59.073740 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:59.107555 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:30:59.108361 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:30:59.299099 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:30:59.319947 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:30:59.320104 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:59.326499 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:59.339792 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:59.354379 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:59.372128 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:30:59.382874 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:30:59.383952 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:59.386667 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:30:59.394239 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:59.407925 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:59.408931 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:59.421890 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:59.465492 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:59.472568 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:59.473582 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:30:59.513530 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:30:59.628927 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:30:59.732554 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:30:59.750422 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:30:59.760476 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:59.801141 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:59.810002 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:59.811332 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:59.813448 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:59.820469 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:30:59.840039 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:30:59.851886 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:30:59.873607 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:30:59.874069 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:30:59.899344 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:30:59.921113 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:30:59.925598 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 14:30:59.941723 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:30:59.941994 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:30:59.970882 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:30:59.972920 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:59.982712 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:30:59.994367 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:30:59.998015 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:00.020117 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:31:00.283924 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:00.305519 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:00.332676 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:00.355643 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:00.356859 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:00.357591 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:00.360383 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:00.360939 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:00.554589 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:31:00.614581 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:31:00.634332 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:31:00.649818 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:00.676074 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:00.685612 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:00.704416 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:31:00.712652 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:00.736205 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:00.755374 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:00.790992 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:00.797797 18900 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.69:41119, user_credentials={real_user=slave}} blocked reactor thread for 71859.2us
W20240502 14:31:00.799578 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:31:00.875049 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:00.931003 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:31:00.972209 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:00.982627 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:00.996722 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:31:00.999444 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:31:01.005326 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:31:01.025408 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:01.039808 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:31:01.040082 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:01.047180 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:01.052486 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:31:01.075253 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:01.102838 20562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:31:01.132781 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:31:01.178753 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:31:01.240725 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:01.258829 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:01.260470 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:01.262372 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:01.275440 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:01.281682 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:01.295019 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:01.329433 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:01.361176 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:31:01.614993 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:01.621172 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:01.679009 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:01.683470 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:01.702499 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:01.705639 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:01.735736 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:01.745036 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:01.755283 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:01.766058 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:01.776422 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:01.788937 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:31:01.794945 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:01.806629 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:31:01.823837 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:01.844403 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:31:01.914887 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:01.921164 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:31:01.986815 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:31:02.001645 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:31:02.060101 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:02.069075 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:02.074522 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:02.084267 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:02.086802 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:02.088858 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:02.186945 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:31:02.196194 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:02.220853 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:31:02.264536 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:02.284191 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:31:02.306969 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 91 similar messages]
I20240502 14:31:02.307106 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:02.309991 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:02.317065 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:31:02.321818 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:02.322870 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:02.332937 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:02.343305 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:31:02.345394 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:02.423899 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:31:02.436105 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:31:02.539395 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:31:02.579324 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:02.584251 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:02.621374 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:02.646582 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:02.651576 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:02.681922 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:02.686550 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:02.823056 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240502 14:31:02.825430 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:02.866807 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:02.874645 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:31:02.885748 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:02.889751 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:02.911435 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:02.913950 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:02.954175 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:03.018468 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:31:03.156978 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:31:03.218389 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:03.227674 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:03.234194 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:03.250500 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:31:03.265046 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:03.267679 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:03.271399 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:03.297053 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:03.315132 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:03.353469 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:31:03.367499 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:03.369283 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:03.373590 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:03.376708 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:03.399495 20964 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.5.107.71:39195, user_credentials={real_user=slave}} blocked reactor thread for 51763.9us
W20240502 14:31:03.400514 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:03.403167 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:03.472128 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:31:03.472978 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:31:03.536721 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:31:03.624917 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:31:03.670519 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:31:03.699296 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:31:03.775962 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 14:31:03.803371 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:31:03.836444 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:03.838765 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:31:03.911131 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:03.912639 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:03.920935 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:03.933439 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:03.935686 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:03.938938 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:31:03.964150 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:03.967197 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:04.060722 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:04.081997 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:04.122311 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:04.122328 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:04.130709 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:04.152909 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:04.273699 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:04.322361 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:31:04.345631 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:04.352445 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:31:04.368945 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:04.417816 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:04.432291 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:04.439354 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:31:04.453334 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:04.455956 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:04.523993 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:31:04.524103 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:31:04.534386 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:04.538341 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:31:04.586637 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:31:04.609757 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:04.614012 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:04.636860 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:04.653434 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:04.667022 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:31:04.669051 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:04.677474 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:04.717859 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:31:04.770159 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:31:04.778494 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:31:04.805208 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:04.855381 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:31:04.872787 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:31:04.910408 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:04.928745 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 14:31:04.931170 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:04.953756 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:04.957782 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:04.965602 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:04.970878 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:31:04.971798 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:05.021574 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:05.114097 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:31:05.139567 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:05.140272 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:05.170665 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:05.172878 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:05.187722 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:05.216073 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:05.336705 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:05.402475 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:31:05.467442 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:05.471717 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:05.496606 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:05.540225 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:05.555361 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:05.578393 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:05.596015 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:05.597879 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:31:05.617389 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:05.656441 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:05.678642 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:05.679387 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:05.709283 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:05.758244 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:31:05.770175 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:05.847919 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:05.850710 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:05.881752 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:31:05.906122 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:31:05.953958 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:31:05.985401 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:05.985569 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:05.992381 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:31:05.997387 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:31:06.014840 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:06.018314 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:06.019634 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:06.020975 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:06.186563 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:31:06.266849 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:06.295109 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:06.302107 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:06.308202 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:06.328730 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:31:06.332835 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:06.340198 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:06.341683 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:06.361126 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:06.396694 20562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:31:06.405478 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:31:06.405826 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:31:06.406116 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:31:06.531625 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:06.555490 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:06.575748 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:06.598786 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:06.605933 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:06.612243 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:06.662402 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:31:06.753096 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:06.765000 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:06.782711 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:06.796766 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:06.797209 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:06.798038 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:06.843219 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:06.859272 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:06.878566 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:31:06.882254 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:31:07.027417 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:07.028455 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:07.039040 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:07.043164 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:07.046128 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:07.059806 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:07.060149 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:31:07.069564 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:31:07.087800 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:31:07.181988 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:31:07.190199 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:31:07.220670 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:31:07.266654 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:31:07.316229 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:31:07.320236 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:07.323287 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:07.334719 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:31:07.339123 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:07.370697 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:07.370967 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:07.387014 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:07.387459 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:07.412925 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 14:31:07.417984 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:07.419848 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:31:07.420379 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:31:07.427790 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:07.444816 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:07.506989 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:31:07.508054 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:07.529539 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:07.582369 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:07.594218 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:07.761541 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:07.844899 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 14:31:07.857800 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:07.871980 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:07.875152 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:07.910683 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:07.921763 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:07.969717 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:07.977576 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:31:08.000247 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:31:08.018224 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:08.029688 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:08.047595 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:08.095563 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:08.115649 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:08.126461 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:31:08.161867 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:08.203853 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:08.208930 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:31:08.228564 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:31:08.339459 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:31:08.347160 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:08.423779 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:08.478747 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:31:08.514393 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:31:08.520989 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:31:08.586336 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:08.626660 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:08.628495 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:08.647243 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:08.652086 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:08.687000 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:08.761137 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:31:08.769786 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:08.770025 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:08.782825 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:31:08.840340 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:08.849793 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:08.854499 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:08.862679 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:08.877089 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:08.892751 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 14:31:08.899848 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:31:08.921700 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:31:08.928700 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:09.115612 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:09.123790 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:09.126828 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:09.128429 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:09.140637 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:09.152716 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:09.175948 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:31:09.222187 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:31:09.309253 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:09.317078 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:09.339242 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:09.342972 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:09.366791 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:09.384436 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:09.426839 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:09.464926 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:31:09.471107 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:09.585925 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:31:09.622714 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:31:09.670126 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:31:09.679646 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:31:09.688958 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:09.690903 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:09.696693 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:31:09.708595 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:09.709000 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:09.722487 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:09.726259 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:09.741047 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:09.758584 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:31:09.761890 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:09.799290 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:09.820626 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:09.842698 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 14:31:09.844964 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:09.845700 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:31:09.851680 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:09.855342 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:31:09.869519 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:09.914685 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:31:09.941269 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:31:09.950631 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:31:09.958094 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 14:31:09.973724 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:31:10.014233 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:10.084841 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:10.092459 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:10.111042 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:10.114403 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:10.118506 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:10.151541 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:10.325311 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:10.370486 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:10.370600 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:10.373574 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:10.381135 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:10.386510 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:10.387984 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:31:10.402464 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:10.579116 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:10.591675 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:31:10.631094 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:31:10.693810 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:10.714488 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:10.732520 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:10.744745 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:31:10.745359 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:31:10.787001 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:10.789855 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:31:10.811131 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:10.816813 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:10.828853 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:10.834890 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:10.858465 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:10.879516 18769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.69:41119, user_credentials={real_user=slave}} blocked reactor thread for 119645us
I20240502 14:31:10.891847 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:10.895745 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:10.903435 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:10.949623 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:31:10.991204 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:11.021565 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:31:11.071405 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:31:11.101858 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:11.137416 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:31:11.201014 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:11.202723 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:11.204000 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:11.234419 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:11.266911 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:31:11.274695 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:11.320677 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:11.362361 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:11.437288 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:31:11.438124 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:31:11.482910 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:11.495245 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:11.496234 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:11.509351 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:11.509469 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:11.523241 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:11.603585 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:11.621659 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:11.747721 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:31:11.752741 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:11.767427 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:11.772334 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:11.776602 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:11.783485 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:11.787364 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:11.859233 20562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 14:31:11.947774 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:31:11.992861 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:31:12.019032 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:12.049305 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:12.071971 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:12.078981 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:31:12.091274 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:12.100350 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:31:12.113516 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:12.135201 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:12.156059 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:12.184677 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:12.210022 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:31:12.213539 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:12.219187 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:12.227839 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:12.251922 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:12.258558 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:12.269454 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:12.273751 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:31:12.285589 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:31:12.332103 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:31:12.397199 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:12.411687 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:31:12.413846 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:31:12.428130 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240502 14:31:12.429625 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:31:12.431954 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:31:12.471079 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:31:12.511544 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:12.530755 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:12.532336 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:12.547080 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:12.555554 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:12.565171 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:31:12.572619 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:12.809804 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:12.838681 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:12.864950 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:12.877400 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:12.929296 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:12.951885 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 14:31:12.956404 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:12.970180 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:31:12.985169 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:12.985651 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:12.986081 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:13.035426 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:31:13.046751 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:13.073869 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:13.082201 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:13.109860 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:13.149871 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:13.209784 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:31:13.229002 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:31:13.314998 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:31:13.333415 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:13.339030 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:13.344588 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:31:13.357537 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:13.357558 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:13.360852 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:13.362828 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:13.363926 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:13.371536 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:13.421619 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:31:13.559643 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:31:13.587932 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:31:13.605547 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:13.619231 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:13.625569 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:13.640127 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:13.648140 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:13.648407 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:13.731182 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:31:13.739584 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:13.754684 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:31:13.866597 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:31:13.869490 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:13.893122 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:13.920691 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:13.923259 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:13.951022 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:13.962258 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:13.964802 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:31:13.966722 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:14.084340 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:31:14.093274 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 63 similar messages]
I20240502 14:31:14.145928 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:14.160794 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:14.165575 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:14.168079 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:14.172636 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:14.203159 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:14.229423 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:14.365756 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:31:14.384114 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:14.420447 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:14.427613 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:14.432230 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:14.435184 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:14.448765 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:14.473910 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:14.510432 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:31:14.576222 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:31:14.594236 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:31:14.602610 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:14.608949 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:14.609431 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:14.629482 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:14.631841 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:14.633775 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:31:14.635576 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:14.668092 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:14.684783 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:31:14.706187 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:31:14.714395 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:31:14.781606 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:31:14.806989 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:31:14.819242 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:14.871420 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:31:14.896409 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:14.903091 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:14.919052 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:14.921030 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:14.924538 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:14.946782 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:15.000598 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:31:15.058204 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 14:31:15.063130 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 14:31:15.064057 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:15.103092 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:31:15.143170 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:31:15.185627 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:15.191416 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:15.226109 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:15.233968 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:15.235023 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:15.235735 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:15.277766 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:31:15.396716 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:15.404054 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:15.407929 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:15.424126 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:15.431712 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:15.434404 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:15.447142 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:15.450315 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:31:15.475363 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:15.647541 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:31:15.648655 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:15.655329 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:15.669946 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:15.679005 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:15.712323 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:15.727277 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:15.737972 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:31:15.745693 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:31:15.766763 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:31:15.799841 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:15.855078 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:31:15.945456 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:15.948845 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:15.983820 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:15.991614 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:16.012916 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:16.024302 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:16.039086 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:16.043015 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:16.065793 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:16.093662 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:16.133702 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:31:16.138095 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:16.138137 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:16.147593 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:16.269156 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:16.278812 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:31:16.297600 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:31:16.299432 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:31:16.325356 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:31:16.355729 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:16.362368 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:16.369453 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:16.371845 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:16.376855 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:16.397822 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:16.450268 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:16.523986 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:31:16.585050 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:16.603498 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:31:16.605307 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:16.605950 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:16.610478 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:16.616662 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:16.633131 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:16.650566 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:16.884341 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:16.891006 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:31:16.897615 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:16.929373 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:16.940579 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:16.951371 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:16.955741 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:16.968151 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:31:16.975504 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:31:17.013473 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:17.039606 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:17.068783 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:31:17.111100 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:31:17.126902 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:17.130107 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:17.132079 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:31:17.136847 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:31:17.137336 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:31:17.140936 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:31:17.148099 19030 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.65:36749, user_credentials={real_user=slave}} blocked reactor thread for 47254.4us
W20240502 14:31:17.172861 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:31:17.173588 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:17.179402 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 14:31:17.182188 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:17.185035 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:17.285269 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:17.358255 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:17.417760 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:17.418378 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:17.462539 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:17.491752 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 93 similar messages]
I20240502 14:31:17.493211 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:17.502090 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:17.506263 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:17.510227 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:31:17.577251 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:31:17.598414 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:31:17.626329 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:17.648530 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:31:17.670231 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:17.674468 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:17.676662 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:17.702132 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:17.726351 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:17.728520 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:17.862442 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:31:17.904004 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:31:17.940017 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:17.956756 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:17.968768 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:17.979447 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240502 14:31:17.994268 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:17.999217 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:18.021881 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:18.039746 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:18.055466 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:18.086369 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:31:18.196048 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:31:18.258071 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:18.269933 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:31:18.304980 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:18.305279 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:18.313436 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:18.330217 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:18.344283 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:18.372537 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:18.400848 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:18.417294 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:18.455951 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:18.461879 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:18.484237 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:31:18.485841 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:18.493887 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:18.495589 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:31:18.502547 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:31:18.614410 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:18.673288 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:31:18.673897 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:18.683588 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:18.702728 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:18.703385 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:18.715260 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:18.722205 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:18.770313 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:31:18.801837 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:18.836948 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:31:18.870893 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:31:18.965781 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:19.001756 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:19.027194 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:19.034426 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:19.043454 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:19.044376 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:19.053659 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:19.096284 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:19.105417 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:19.146596 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:19.149784 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 14:31:19.179538 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:31:19.189604 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:19.190171 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:19.226195 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:19.267392 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:31:19.288637 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:31:19.332774 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:19.421638 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:31:19.433655 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:19.434151 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:19.452315 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:31:19.452941 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:19.458833 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:19.463490 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:19.467056 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:31:19.473857 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:19.475656 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:31:19.494679 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:31:19.575080 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:19.677342 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:19.683319 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:19.695930 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:19.698287 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:19.698585 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:19.706650 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:19.725912 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:31:19.739478 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:31:19.739593 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:31:19.740075 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:31:19.823205 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:19.904628 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:31:19.976365 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:19.989431 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:19.990403 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:31:20.007263 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:20.014021 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:20.047401 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:20.054709 19032 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.69:41119, user_credentials={real_user=slave}} blocked reactor thread for 58097us
W20240502 14:31:20.058916 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 14:31:20.082767 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:20.095469 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:20.137639 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:20.141366 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:20.145224 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:20.174616 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:31:20.176311 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:20.184244 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:20.188459 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:20.240456 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:31:20.259006 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:31:20.283387 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:31:20.284269 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:20.298533 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:31:20.427577 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:20.447500 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:31:20.468254 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:20.472496 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:20.474659 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:20.487386 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:20.492600 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:20.509306 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:20.553398 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:20.594326 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:20.597273 18637 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.71:39195, user_credentials={real_user=slave}} blocked reactor thread for 149783us
I20240502 14:31:20.637001 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:20.637405 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:20.640987 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:20.671200 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:31:20.675840 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:20.697079 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:31:20.741119 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:20.840314 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:31:21.019124 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:31:21.107784 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:21.111368 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:31:21.138779 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:31:21.142131 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:21.146812 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:21.178469 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:21.187712 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:21.197552 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:21.218106 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:31:21.218232 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:21.241868 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:21.241956 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:21.266443 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:31:21.277004 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:21.309368 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:21.338995 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:21.344347 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:21.365274 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:21.648469 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:21.654459 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:21.705691 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:21.727174 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:21.743486 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:21.757445 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:21.784010 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:21.789294 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:21.794107 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:21.814195 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:31:21.827492 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:21.847659 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:31:21.852941 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:21.862140 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:21.886139 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:21.898114 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:31:21.904666 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:21.930212 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:31:21.946235 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:31:21.962345 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:31:22.035609 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:31:22.046239 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:31:22.111625 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:31:22.126073 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:31:22.146407 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:22.151480 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:22.153363 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:22.167200 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:22.189121 20562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 14:31:22.196877 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:22.204463 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:22.267628 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:31:22.276057 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:31:22.284238 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:22.335844 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:22.388444 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:31:22.401436 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:31:22.425822 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:31:22.443717 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:22.457111 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:22.461941 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:22.468365 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:22.484921 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:22.507316 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:22.521086 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 14:31:22.619549 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:31:22.640827 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:22.692603 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:31:22.695806 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:31:22.745878 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:22.756024 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:22.757552 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:22.773427 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:31:22.832144 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:22.836007 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:22.845705 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:22.875231 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:22.905459 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:31:22.972826 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:31:23.020501 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240502 14:31:23.025974 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:23.033097 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:23.037452 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:23.073268 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:23.082460 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:23.096596 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:23.106001 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:23.207018 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:31:23.265413 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:23.275589 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:23.287451 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:23.301730 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:23.317687 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:23.320922 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:23.346089 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:23.375308 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:31:23.425549 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:31:23.509697 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:31:23.545527 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:31:23.633517 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:31:23.640174 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:23.658777 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:23.685832 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:23.698954 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:23.712146 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:23.724586 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:23.729276 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:23.754209 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:23.767668 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:23.799222 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:23.802098 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:31:23.850801 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:23.852929 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:23.889410 18769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.68:41463, user_credentials={real_user=slave}} blocked reactor thread for 45313us
W20240502 14:31:23.896704 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:23.901971 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:23.941466 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:31:24.095052 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:24.104777 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:31:24.108460 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:24.136808 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:24.144840 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:24.149749 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:24.179302 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:24.179930 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:24.248287 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:31:24.267408 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 14:31:24.320751 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:31:24.383822 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:24.397634 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:24.406286 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:31:24.411759 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:24.412003 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:24.440506 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:24.445955 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:24.479176 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:24.503407 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:24.505733 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:31:24.535349 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:31:24.575582 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:31:24.640760 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:31:24.694799 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:31:24.703576 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:24.707607 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:31:24.708717 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:31:24.725674 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:24.735069 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:24.741988 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:31:24.766211 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:24.769342 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:24.781443 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:24.873601 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:31:24.951606 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:24.954912 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:24.957124 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:24.999627 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:25.005112 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:25.009610 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:25.053608 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:25.055665 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:31:25.075287 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:31:25.101009 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:25.116930 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:31:25.137149 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:31:25.182250 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:25.199395 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:25.202354 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:31:25.236747 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 14:31:25.251787 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:25.256819 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:25.278023 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:25.298128 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:25.431948 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:31:25.435865 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:25.449573 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:31:25.455873 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:25.467121 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:25.472095 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:25.506250 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:25.507266 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:25.547884 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:25.633376 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:31:25.680069 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:31:25.682121 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:25.737370 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:25.744964 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:25.751312 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:25.751926 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:25.751945 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:25.760845 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:25.807662 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:31:25.902757 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:31:26.000305 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:31:26.070899 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:26.075631 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:26.088160 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:26.090451 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:31:26.115144 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:26.116108 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:26.121119 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:26.136351 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:26.143440 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:31:26.149158 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:26.162618 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:26.178009 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:31:26.191941 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:26.192266 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:26.209209 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:26.260248 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:26.286531 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:26.431635 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:31:26.459975 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:26.470631 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:26.481464 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:26.486855 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:26.487891 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:26.504766 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:26.708416 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:31:26.736609 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:26.747504 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:26.799664 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:26.819568 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:31:26.843112 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:26.849658 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:26.861218 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:31:26.861335 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:31:26.875326 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:26.878469 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:31:26.894179 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:26.905220 20964 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.5.107.71:39195, user_credentials={real_user=slave}} blocked reactor thread for 54386.6us
W20240502 14:31:26.919093 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:26.973510 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:26.977432 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:31:26.990154 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:27.002744 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:31:27.043246 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:27.068342 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:27.093487 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:27.104368 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:27.182605 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:27.185393 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:31:27.233310 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:31:27.266492 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:31:27.304787 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:27.307966 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:27.321421 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:27.334728 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:27.341246 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:31:27.343593 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:27.343587 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:27.354493 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:31:27.387194 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:27.474834 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:31:27.486907 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:31:27.524854 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:31:27.541004 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:31:27.560297 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 88 similar messages]
I20240502 14:31:27.604897 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:27.613862 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:27.619784 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:27.624100 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:27.632656 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:27.647490 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:27.723897 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:31:27.786991 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:27.807281 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:31:27.841001 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:27.876240 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:31:27.885818 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:27.919561 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:27.949095 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:27.965754 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:27.966559 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:27.978843 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:28.019938 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:31:28.023948 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240502 14:31:28.035854 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:28.056641 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:28.087284 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:28.102403 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:31:28.109145 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:28.117621 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:28.125948 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:28.193382 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:31:28.275849 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:31:28.290293 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:28.361255 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:28.375301 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:28.380945 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:31:28.382144 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:28.397292 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:28.399130 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:28.403780 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:31:28.408288 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:28.470856 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:28.553920 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:28.554621 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:28.592305 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:31:28.604796 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:28.618094 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:28.622898 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:28.637398 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:28.702066 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:28.709137 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:31:28.870580 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:28.877729 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:31:28.904306 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:28.922969 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:28.943039 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:28.951728 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:28.953874 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:31:28.980314 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:29.006516 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:29.015307 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:31:29.192472 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:31:29.230182 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:29.245090 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:29.248493 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:29.251255 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:29.266055 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:29.270226 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:29.277400 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:31:29.290746 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:29.312155 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:29.318063 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 14:31:29.378034 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:31:29.456946 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:29.479209 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:29.505808 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:29.520233 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:29.524381 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:29.549568 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:31:29.552773 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:29.566565 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:31:29.611555 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:31:29.612525 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:31:29.661720 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:31:29.696128 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:29.730810 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:29.748317 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:29.755715 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:29.783915 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:29.805994 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:29.824978 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:29.835454 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:29.883405 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:31:29.906085 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:31:29.941984 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:31:30.054097 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:31:30.090135 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:31:30.155318 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:30.179906 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:30.222011 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:30.225291 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:31:30.225587 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:30.237167 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:31:30.246234 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:30.258916 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:30.304168 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:30.319414 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:30.324406 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:30.363448 19032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 14:31:30.414291 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:31:30.426541 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:30.446552 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:30.451232 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:31:30.483902 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:30.505232 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:30.519542 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:30.530076 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:31:30.542548 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:31:30.590391 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:31:30.712203 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:30.715348 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:30.723484 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:30.725370 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:30.736407 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:31:30.740590 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:30.742398 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:30.743584 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:30.776137 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:31:30.831003 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:31:30.831918 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:30.950685 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:31:30.983696 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:30.995463 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:31.000703 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:31.009593 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:31.033084 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:31.077888 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:31.163549 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:31:31.272336 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:31.300518 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:31.303737 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:31.310775 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:31.340970 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:31.341668 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:31.345358 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:31:31.356563 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:31.360260 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:31.452236 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:31:31.519551 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:31.526551 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:31.532990 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:31.544696 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:31.564375 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:31.599133 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:31.641471 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:31:31.647795 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:31:31.688323 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:31:31.693477 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:31:31.776345 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:31.787147 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:31.796386 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:31.814726 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:31.846438 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:31.851790 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:31.910609 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:31.927711 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:31:31.934954 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:31:31.934935 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:31.984407 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:31:32.047204 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:32.061607 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:32.065032 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:32.065078 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:32.065659 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:32.089138 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:32.154505 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:31:32.188758 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:31:32.229506 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:31:32.287756 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:32.344115 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:31:32.354640 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:32.384446 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:32.392565 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:32.396561 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:32.401621 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:32.419436 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:32.431398 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:32.504736 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:31:32.540568 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:31:32.624487 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240502 14:31:32.639984 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:31:32.642812 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:31:32.712523 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:32.716022 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:32.729074 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:32.746574 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:31:32.761091 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:32.766396 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:32.767493 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:32.774927 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:32.845937 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:32.849875 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:31:32.921973 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:32.937804 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:32.959415 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:32.965632 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:32.970009 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:32.982918 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:32.985605 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 14:31:33.012830 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:31:33.024412 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 76 similar messages]
W20240502 14:31:33.025511 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:31:33.092957 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:31:33.120677 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:31:33.206244 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:33.214272 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:33.215500 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:33.229122 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:33.229607 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:33.229872 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:33.243556 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:31:33.276918 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:31:33.346164 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:33.380329 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:33.514653 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:31:33.539135 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:33.550307 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:33.568607 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:33.585721 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:33.592568 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:31:33.596830 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:33.619939 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:33.628434 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:33.631381 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:31:33.654445 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:33.677284 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:33.747704 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:33.751859 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:33.769624 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:33.783167 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:33.788295 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:33.931104 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:31:33.954173 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:31:33.991196 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:33.994824 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:34.010506 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:34.011210 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:34.021497 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:34.056196 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:34.148684 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:31:34.160980 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:31:34.218963 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:31:34.259986 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:34.266600 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:34.266984 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:34.270087 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:34.271267 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:31:34.290730 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:34.296146 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:34.324754 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 14:31:34.339182 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:31:34.341708 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:34.347489 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:34.403560 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:31:34.455626 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:31:34.583864 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:34.594350 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:34.617098 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:31:34.622442 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:34.630120 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:34.675649 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:31:34.676829 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:34.713635 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:34.752568 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:34.765992 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:34.835433 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:34.859133 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:31:34.879498 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:34.894129 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:34.894294 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:34.898587 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:31:34.909938 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:34.936446 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:34.985527 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:31:35.021800 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:31:35.037772 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:31:35.093254 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:31:35.124301 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:35.147950 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:35.153656 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:31:35.157747 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:35.158116 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:35.164620 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:35.199400 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:35.220824 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:31:35.294380 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:35.334519 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:35.337990 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:35.343937 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:35.349381 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:35.351660 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:35.360616 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:35.375126 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:31:35.396715 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:35.527868 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 14:31:35.582983 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:31:35.583616 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:31:35.614830 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:31:35.652000 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:35.680408 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:35.698413 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:35.701853 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:35.714778 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:35.721333 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:35.778077 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:35.787259 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:31:35.827348 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:35.890753 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:31:35.947703 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:31:35.968364 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:35.979816 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:35.999202 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:36.005503 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:36.021981 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:36.046520 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:36.123538 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:31:36.179440 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:36.197393 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:36.206287 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:36.213744 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:31:36.214896 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:36.240409 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:31:36.251935 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:36.253917 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:36.289397 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:36.319242 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:36.460868 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:36.492511 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:36.497709 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:36.512357 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:36.512926 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:36.516414 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:36.516938 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:31:36.521528 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:31:36.551460 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:31:36.581384 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:31:36.581527 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:31:36.609473 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:36.738001 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:36.790573 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:31:36.837692 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:36.850066 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:31:36.862231 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:36.874598 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:36.887923 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:36.898558 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:36.922580 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:36.928309 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:36.932420 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:36.977514 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:37.002990 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:37.019100 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:37.043272 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:37.053048 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:31:37.089996 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:31:37.103749 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:31:37.112392 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:37.135138 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:31:37.210271 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:31:37.276247 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:37.285213 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:37.297044 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:37.309077 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:37.316241 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:37.339408 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:31:37.353111 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:37.361843 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:37.410701 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:31:37.443361 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:31:37.506572 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:31:37.526381 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:37.527508 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:37.550055 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:37.560359 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:37.569231 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:31:37.570603 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:37.581563 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:37.598541 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:37.760735 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 14:31:37.762504 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:31:37.765908 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:31:37.883949 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:31:37.885874 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:37.928702 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:37.943652 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:37.946205 20964 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.5.107.71:39195, user_credentials={real_user=slave}} blocked reactor thread for 66132.9us
W20240502 14:31:37.952988 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:31:37.952708 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:37.962512 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:37.998983 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:38.001462 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:38.015924 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:38.046777 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 81 similar messages]
I20240502 14:31:38.053926 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:38.055109 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:38.065651 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:38.069487 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:38.099982 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:38.143209 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:38.306777 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:38.350936 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:31:38.351918 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:31:38.377604 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:31:38.404867 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:38.406950 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:31:38.434336 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:38.440570 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:38.457999 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:31:38.486443 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:38.488842 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:38.504552 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:31:38.506750 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:38.516180 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:38.537950 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:31:38.543062 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:38.631063 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:31:38.635936 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:38.663688 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:38.664867 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:38.686455 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:38.690299 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:38.729820 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:38.860188 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:31:38.972857 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:31:39.033350 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:31:39.044031 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:31:39.083112 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:39.097393 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:39.105958 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:39.117960 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:39.122131 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:39.128542 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:39.143908 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:39.147707 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:31:39.164217 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:39.284682 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:31:39.298060 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:39.321295 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:39.324301 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:31:39.343695 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:39.348915 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:31:39.368153 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 14:31:39.374310 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:39.387096 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:39.393484 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:39.417413 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:31:39.501048 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:39.603580 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:39.612047 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:39.628386 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:39.628706 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:31:39.644223 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:39.647735 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:31:39.653149 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:39.687338 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:39.705624 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:31:39.708712 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:31:39.737350 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:31:39.775997 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:39.848841 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:39.853595 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:39.856065 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:39.864985 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:39.866266 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:39.871879 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:39.961941 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:31:39.989326 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:31:40.031857 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:40.078910 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:31:40.165170 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:31:40.170881 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:40.202351 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:40.212018 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:40.252952 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:40.265215 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:40.297609 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:40.309540 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:40.339526 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:40.349718 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:40.362972 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:40.377972 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:40.385250 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:40.396842 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:40.419744 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:31:40.490130 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:31:40.551430 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:40.651677 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:40.652743 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:40.659041 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:31:40.672909 19032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 14:31:40.677006 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:40.693389 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:40.705253 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:40.708966 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:40.714552 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:40.804996 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:31:40.873211 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:31:40.907953 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:40.917726 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:40.920236 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:40.926007 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:40.926944 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:40.943665 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:31:40.949625 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:40.967053 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:31:41.015111 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:31:41.063154 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:31:41.105772 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:41.126166 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:41.190546 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:41.191905 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:31:41.222501 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:41.270186 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:41.281541 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:31:41.281952 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:41.288898 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:41.330510 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:41.352798 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:41.357452 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:41.401818 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:41.412258 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:41.420176 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:41.439877 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:41.462054 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:31:41.474732 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:31:41.587054 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:31:41.589984 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:31:41.657100 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:31:41.661823 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:31:41.707360 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:31:41.708467 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:41.716553 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:31:41.731168 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:41.748157 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:41.756847 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:41.758574 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:41.772624 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:41.773738 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:41.828619 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:41.863343 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:31:41.919361 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:41.926541 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:41.936117 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:41.949457 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:41.964026 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:41.966502 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:42.032276 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:31:42.146492 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:31:42.189621 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:31:42.199404 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:42.210897 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:42.294695 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:31:42.347162 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:42.362457 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:42.373728 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:42.397349 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:42.404811 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:42.410156 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:31:42.452783 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:42.454341 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:31:42.502300 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:31:42.524454 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:31:42.561410 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:31:42.633855 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:42.635282 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:42.642355 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:42.651278 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:42.680392 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:42.695953 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:42.700381 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:42.765137 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 14:31:42.786012 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:42.879455 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:42.887938 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:42.908228 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:42.908412 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:42.911072 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:42.936503 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:43.010006 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:31:43.143277 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:43.158715 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 14:31:43.185633 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:31:43.189074 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:43.202025 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:31:43.205096 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:31:43.207309 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:43.216526 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:43.243544 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:43.253863 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:43.268411 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:43.276368 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:31:43.291438 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:43.312675 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:31:43.447862 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:43.460834 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:43.469904 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:43.480302 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:43.484519 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:43.491137 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:43.501767 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:31:43.507633 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:31:43.578186 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:43.609012 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:31:43.632544 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:31:43.674957 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:43.703387 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:31:43.726740 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:43.735608 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:43.743468 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:43.748039 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:43.752456 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:43.761315 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:43.766151 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:31:43.887197 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:31:44.033327 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:44.059566 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:44.064567 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:44.069476 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:44.071009 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:44.080348 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:44.090399 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:44.091589 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:44.097874 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:31:44.133410 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:44.161513 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:31:44.188838 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:31:44.200759 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:44.207576 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:44.239611 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:31:44.259132 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:44.261207 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:31:44.285490 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:44.286589 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:44.307255 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:31:44.428576 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:31:44.437523 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:31:44.481881 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:44.485103 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:44.492483 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:44.493366 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:44.508054 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:44.515625 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:44.528231 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:44.587234 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 14:31:44.627578 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:44.694888 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:31:44.697556 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:31:44.746037 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:31:44.847263 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:44.859755 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:44.867394 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:44.870211 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:31:44.872223 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:44.935333 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:44.951195 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:44.989663 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:31:45.002476 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:45.034793 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:45.049461 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:45.050285 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:45.066115 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:45.087219 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:31:45.097268 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:45.098788 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:45.175643 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:31:45.187772 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:45.264570 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:31:45.326251 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:31:45.339917 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:45.346740 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:45.353427 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:45.362834 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:45.365460 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:45.381639 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:45.476398 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:45.518743 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:31:45.528910 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:31:45.575314 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:31:45.643493 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:45.645581 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:45.645768 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:45.660120 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:45.671211 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:45.694738 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:31:45.696610 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:45.728353 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 14:31:45.732406 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:45.739071 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:31:45.835714 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:31:45.908789 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:45.923120 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:45.941381 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:31:45.988662 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:46.000622 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:46.001564 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:46.032734 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:46.067247 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:46.107473 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:31:46.143281 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:46.149706 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:46.176538 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:46.197786 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:46.203248 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:46.229985 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:46.272178 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:31:46.312296 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:46.424345 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:31:46.459414 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:46.482689 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:31:46.485268 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:46.495723 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:46.504364 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:31:46.519809 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:31:46.566202 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:46.581028 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:46.595994 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:46.653769 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:31:46.661725 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:46.664314 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:46.681897 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:31:46.708278 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:46.755241 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:46.760608 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:31:46.762719 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:46.769989 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:31:46.776316 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:46.779261 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:46.779842 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:31:46.805958 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:47.010943 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:31:47.027527 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:31:47.058279 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:31:47.072870 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:47.198437 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:47.206302 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:31:47.223240 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:47.225723 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:47.231835 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:47.234089 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:47.236344 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:47.246734 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:47.331661 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:31:47.367286 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:31:47.388815 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:31:47.439558 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:47.463645 18899 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.72:35871, user_credentials={real_user=slave}} blocked reactor thread for 48716.1us
I20240502 14:31:47.468387 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:47.469076 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:47.480109 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:47.495239 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:47.496471 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:47.498636 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:47.640064 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:31:47.667001 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:31:47.721415 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:47.738304 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:47.744593 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:31:47.745435 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:47.776858 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 88 similar messages]
I20240502 14:31:47.779161 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:47.779817 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:47.803064 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:47.807874 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:31:47.835039 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:47.937474 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:31:47.972229 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:47.992213 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:47.995539 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:48.009310 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:48.012820 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:48.048871 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:48.081377 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:48.133311 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:31:48.206115 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 14:31:48.206305 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:31:48.246963 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:31:48.258072 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:48.290585 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:31:48.311151 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:48.312253 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:48.312788 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:48.350708 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:48.359943 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:48.368599 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:48.441869 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:31:48.507117 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:48.513007 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:48.516383 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:48.517971 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:48.552439 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:48.557901 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:48.584175 20562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:31:48.654692 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:31:48.664685 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:31:48.687953 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:48.730387 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:48.735918 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:48.752162 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:48.762872 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:48.765333 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:48.775518 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:48.834941 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:31:48.873937 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:49.001081 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:31:49.023409 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:49.023723 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:49.032305 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:49.039031 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:49.046546 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:49.061764 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:31:49.063086 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:49.093237 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:31:49.114861 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:31:49.169770 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:31:49.203553 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:31:49.229337 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:49.322877 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:31:49.336200 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:49.385828 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:31:49.386066 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:49.418483 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:49.446928 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:49.457865 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:49.464201 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:49.468046 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:49.509917 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:49.525314 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:49.538561 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:31:49.551990 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:49.572683 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:49.593938 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:49.620631 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 14:31:49.629860 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:31:49.630228 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:49.744611 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:31:49.763269 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:31:49.824054 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:49.855077 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:31:49.900965 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:49.906010 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:49.911569 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:31:49.939271 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:31:49.954290 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:49.973477 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:49.979573 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:49.992245 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:50.020023 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:31:50.047274 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:50.097676 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:31:50.143061 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:50.150095 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:50.152732 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:31:50.182340 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:50.207922 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:50.208050 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:50.209439 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:50.313668 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:50.317425 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:31:50.427239 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:50.467020 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:31:50.549715 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:50.578011 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:31:50.578164 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:50.582060 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:50.592572 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:50.603873 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:50.669027 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:31:50.682428 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:50.691141 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:31:50.727648 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:31:50.729805 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:31:50.846048 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 14:31:50.874788 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:50.876121 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:50.898265 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:50.916561 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:50.926121 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:50.967105 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:50.985113 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:51.020694 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:51.089651 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:51.106730 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:51.131579 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:51.132548 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:51.181974 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:51.203382 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:51.278949 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:31:51.297684 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:51.297873 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:31:51.362143 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:31:51.384228 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:51.422768 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:51.455531 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:51.465495 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:51.470912 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:51.482148 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:51.486927 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:51.518708 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:31:51.518862 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:31:51.560317 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:31:51.611253 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:31:51.627538 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:31:51.731012 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:31:51.750746 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:51.798882 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:51.803017 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:51.814288 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:31:51.815574 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:51.815771 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:51.820984 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:51.824479 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:51.839057 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:51.870245 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:31:51.912252 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:51.923452 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:31:51.939007 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:51.957830 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:31:52.004379 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:52.012077 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:52.036300 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:52.037603 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:52.071238 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:31:52.131493 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:31:52.173525 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:31:52.229032 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:52.238996 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:31:52.270821 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:52.281870 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:31:52.295958 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:31:52.315888 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:52.316823 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:52.342415 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:52.343071 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:52.354187 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:52.404194 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:31:52.442726 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:52.457283 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:31:52.610086 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:52.611491 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:52.617014 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:52.628329 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:52.637465 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:52.657891 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:52.678939 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:52.758806 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:31:52.759099 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:31:52.779135 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 14:31:52.886731 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:31:52.900447 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:52.901024 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:52.935186 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:52.966506 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:52.966507 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:52.988931 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:53.005673 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:53.022891 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:53.032438 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:53.129885 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:31:53.135988 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:53.136344 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:53.139079 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:53.158921 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:31:53.163123 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:53.229503 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:31:53.238659 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:53.269809 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:31:53.276798 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:53.333125 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 81 similar messages]
I20240502 14:31:53.428822 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:53.436411 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:53.438189 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:31:53.456640 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:53.464555 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:53.464852 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:53.469426 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:53.484395 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:53.485986 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:53.508508 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:53.554996 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:53.569540 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:53.610802 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:53.689004 20562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:31:53.694509 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:53.787801 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:31:53.855266 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:31:53.856619 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:53.875927 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:31:53.909999 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:31:54.019768 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:31:54.033725 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:31:54.097280 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:54.104291 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:54.112430 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:54.121814 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:31:54.148231 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:54.153245 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:54.172116 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:54.227197 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:54.240231 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:31:54.248566 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:54.249235 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:54.249562 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:54.268029 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:54.294131 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:31:54.326942 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:54.345695 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:54.346073 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:31:54.385381 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:54.514420 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:54.556598 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:31:54.569468 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:31:54.614351 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:31:54.644076 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:31:54.653165 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:31:54.673542 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:54.678205 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 14:31:54.679428 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:31:54.691179 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:31:54.692112 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:54.705998 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:54.715319 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:54.735535 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:54.740236 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:54.793437 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:54.880254 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:31:54.904649 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:31:54.927599 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:54.949703 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:54.949720 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:54.949703 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:54.976861 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:54.980427 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:54.981860 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:55.039005 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:31:55.287947 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:55.292373 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:31:55.296746 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:31:55.327559 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:55.337251 18900 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.69:41119, user_credentials={real_user=slave}} blocked reactor thread for 44361.1us
I20240502 14:31:55.342048 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:55.354218 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:55.379359 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:55.379426 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:55.387888 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:55.395251 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:55.415182 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:55.416251 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:31:55.464860 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:55.492242 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:55.510186 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:55.529038 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:55.597687 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:55.635231 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:31:55.638782 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:31:55.733875 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:55.736577 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:31:55.738570 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:55.743546 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:55.751603 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:55.764328 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:55.770740 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:31:55.785871 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:55.855473 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:55.975642 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:56.012256 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:56.026528 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:56.027107 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:56.041524 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:56.042018 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:56.065821 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:56.079411 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 14:31:56.080968 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:31:56.160032 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:31:56.335628 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:56.337632 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:56.340514 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:56.354719 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:31:56.365844 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:56.392757 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:31:56.401439 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:56.407945 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:56.408881 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:56.410454 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:56.428289 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:31:56.446013 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:56.449947 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:56.499282 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:31:56.502140 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:56.504688 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:31:56.526633 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:56.546680 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:31:56.553730 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:56.578058 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:56.690953 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:31:56.806200 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:56.823556 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:56.844504 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:56.845402 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:56.857263 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:56.889891 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:56.900523 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:31:56.907872 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:56.942137 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:31:56.942317 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:31:56.951248 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:56.951885 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:56.956594 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:56.974027 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:31:56.975128 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:56.986815 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:31:56.999416 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:57.005980 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:57.007812 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:57.029469 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:31:57.130676 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:31:57.155256 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:31:57.242297 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:31:57.299980 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:31:57.318200 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:31:57.341742 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:31:57.422372 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:57.441566 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:31:57.586429 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:57.590054 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:31:57.603885 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:31:57.604346 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:57.616468 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:57.620033 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:57.620062 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:57.627223 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:57.633574 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:57.652290 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:57.743372 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:31:57.764447 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:57.773758 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:57.799770 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:57.803571 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:31:57.804924 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:57.829483 19031 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.72:35871, user_credentials={real_user=slave}} blocked reactor thread for 55334.9us
W20240502 14:31:57.888855 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240502 14:31:57.993886 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:58.065208 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:31:58.071902 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:58.083086 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:58.089164 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:31:58.099645 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:58.099934 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:31:58.105623 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:58.106809 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:58.111961 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:58.260425 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:58.295507 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:58.309003 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:58.310467 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:58.312258 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:58.312606 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:31:58.352833 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:58.392647 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 14:31:58.421276 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:58.493542 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:31:58.536039 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:31:58.548686 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:58.556752 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:58.576670 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:58.587677 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:58.592221 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:31:58.610818 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:58.611708 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:58.693586 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:31:58.701206 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:31:58.864367 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:58.864614 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:58.867805 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:58.876286 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:58.879292 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:58.888463 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 14:31:58.905519 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:58.906001 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:58.919224 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:31:58.931515 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:31:58.981575 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:31:58.986812 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:59.072901 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:31:59.086233 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:31:59.135951 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:31:59.221426 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:59.226601 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:59.279840 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:59.281090 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:59.295615 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:59.319221 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:31:59.326524 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:59.346761 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:59.373493 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:59.403896 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:59.426151 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:31:59.442059 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:31:59.467254 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:59.471801 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:31:59.485488 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:31:59.505288 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:31:59.522082 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:31:59.541571 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:31:59.564913 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:31:59.691546 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:31:59.694893 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:31:59.712450 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:31:59.714323 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:59.715550 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:31:59.715929 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:31:59.716264 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:31:59.725617 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:31:59.752514 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:31:59.798878 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 14:31:59.890784 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:31:59.955425 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:31:59.969434 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:32:00.003307 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:00.041661 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:32:00.046408 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:32:00.051632 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:00.053527 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:00.064103 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:00.076659 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:00.087306 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:00.097218 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:00.140686 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:00.145593 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:00.196992 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:32:00.201437 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:00.205861 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:00.219058 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:00.221400 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:32:00.235360 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:00.306293 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:00.380869 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:32:00.501102 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:32:00.504123 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:00.511296 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:00.517568 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:00.522044 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:00.534862 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:00.553231 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:00.597132 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:00.675721 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:32:00.701329 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:00.797416 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:00.802580 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:32:00.805439 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:00.807269 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:00.829946 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:00.835016 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:00.842456 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:01.007192 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:01.115275 19032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 14:32:01.116705 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:32:01.149834 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:01.153522 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:01.158874 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:32:01.171147 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:01.176115 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:01.188330 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:01.198590 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:32:01.203387 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:01.209437 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:01.273358 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:01.284456 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:01.286005 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:01.290293 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:01.345468 20964 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.5.107.71:39195, user_credentials={real_user=slave}} blocked reactor thread for 54209.1us
W20240502 14:32:01.348526 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:01.349115 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:01.402869 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:32:01.483021 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:01.531211 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:32:01.532781 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:32:01.589133 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:32:01.612532 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:32:01.645381 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:01.668936 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:32:01.670540 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:01.698725 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:01.734490 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:01.750114 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:01.760293 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:01.767653 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:01.821606 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:01.822993 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:32:01.827140 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:01.857421 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:01.860621 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:01.888531 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:32:01.898016 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:01.922719 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:01.924422 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:32:01.973206 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:32:02.061640 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:02.099754 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:32:02.102142 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:32:02.275615 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:32:02.304778 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:02.309558 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:02.311019 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:32:02.313971 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:02.318540 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:32:02.362352 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:32:02.368079 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:02.373993 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:02.411265 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:02.433238 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:32:02.448153 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:02.498454 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:32:02.545948 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:32:02.564164 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:32:02.602228 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:02.616357 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:02.646863 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:32:02.648658 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:02.673952 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:02.675101 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:02.702786 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:02.706102 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:02.769625 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:32:02.791010 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:02.896818 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:02.922827 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:02.925475 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:02.927922 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:02.932008 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 91 similar messages]
I20240502 14:32:02.949165 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:02.958079 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:03.024935 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:32:03.110374 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:32:03.116979 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:32:03.118928 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:03.168814 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:03.169577 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:32:03.174194 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:03.190873 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:03.194365 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:03.214365 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:03.238085 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:03.372496 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:03.375557 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:03.379498 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:03.386907 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:03.409493 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:03.422708 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:03.445948 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:03.489722 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 14:32:03.502753 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:03.734062 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:03.739673 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:03.755880 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:32:03.757556 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:03.761205 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:32:03.765081 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:03.770828 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:03.780664 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:03.861699 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:32:03.923586 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:03.966642 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:32:04.021908 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:04.040261 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:32:04.045596 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:04.048071 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:04.054889 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:04.055102 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:04.069015 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:04.078794 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:32:04.099674 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:04.101219 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:32:04.119624 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 14:32:04.255362 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:04.286108 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:32:04.294049 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:32:04.295917 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:04.314833 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:04.333014 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:04.338224 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:32:04.357789 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:32:04.367816 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:04.420840 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:32:04.441699 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:04.463639 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:04.538224 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:04.574316 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:32:04.597425 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:04.600603 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:04.604712 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:04.607650 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:32:04.608661 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:04.609549 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:04.615389 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:04.681342 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:32:04.692659 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:32:04.755200 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:32:04.892158 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:04.897210 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:04.904678 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:32:04.910823 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:32:04.917901 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:04.919718 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:04.924206 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 14:32:04.925072 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:32:04.935557 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:04.948000 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:04.949461 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:04.961977 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:05.006639 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:32:05.013059 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:32:05.014205 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:05.075274 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:05.081971 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:05.084424 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:05.106946 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:05.107172 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:32:05.110890 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:05.322855 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:05.327685 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:05.331775 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:05.354795 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:05.355587 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:05.356953 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:05.363747 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:32:05.414232 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:32:05.428678 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:05.561540 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:05.568261 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:05.594285 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:05.601347 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:05.604765 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:05.613852 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:32:05.615772 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:05.667586 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:32:05.680967 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:05.788990 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:32:05.888816 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:05.916985 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:05.947573 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:32:05.978154 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:05.991927 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:06.012856 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:06.016531 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:32:06.058430 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:06.114533 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:06.155416 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:06.157106 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 14:32:06.171837 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:06.203346 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:06.239686 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:06.245898 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:06.273552 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:06.318014 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:32:06.320288 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:06.328760 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:32:06.386137 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:06.489388 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:32:06.508181 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:32:06.555835 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:06.564342 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:06.575569 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:06.589437 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:06.593727 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:06.607944 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:32:06.613864 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:06.660419 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:32:06.733031 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:32:06.757344 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:06.764214 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:06.770280 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:06.790278 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:06.805500 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:06.809329 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:06.828502 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:32:06.830036 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:06.848664 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:06.849669 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:32:06.891469 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:32:06.892768 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:32:07.089485 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:32:07.127319 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:32:07.130544 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:07.146348 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:07.158778 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:07.185274 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:32:07.186314 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:07.187692 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:07.219749 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:07.239318 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:32:07.252940 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:07.329962 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:32:07.372021 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:32:07.432945 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:32:07.443225 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:07.478912 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:07.479658 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:07.510372 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:32:07.558539 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:07.559574 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:07.592576 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:07.598040 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:32:07.599841 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:07.662714 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:32:07.667194 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:07.681728 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:07.709224 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:32:07.733218 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:07.749620 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:07.750380 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:07.760186 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:07.783483 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:07.786718 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:32:07.871640 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:32:07.950188 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 91 similar messages]
I20240502 14:32:07.955964 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:07.961653 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:07.965236 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:07.968459 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:32:07.985929 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:07.986349 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:07.986666 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:08.034518 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:08.086401 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:32:08.231473 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:08.239610 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:08.261144 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:08.281368 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:08.282613 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:08.304224 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:08.342242 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:08.370077 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:32:08.373346 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:32:08.416536 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:08.423635 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:08.442838 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:08.444295 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:08.453413 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:08.469491 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:08.490255 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:08.497706 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240502 14:32:08.658882 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:08.664466 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:08.680397 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:08.690199 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:08.694103 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:08.712323 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:32:08.714318 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:08.849901 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:32:08.873844 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:08.880448 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:32:08.926869 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:08.929881 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:08.951532 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:08.977030 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:32:08.979312 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:09.000049 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:09.016080 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:09.068718 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:09.075901 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:32:09.116458 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:32:09.118674 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:32:09.218919 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:32:09.251561 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:09.256407 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:32:09.279460 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:09.279879 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:09.292794 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:32:09.299475 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:09.318115 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:09.342406 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:09.382953 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:09.386919 20562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:32:09.412452 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:32:09.432325 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:32:09.436923 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:09.468868 18768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.72:35871, user_credentials={real_user=slave}} blocked reactor thread for 49937.4us
I20240502 14:32:09.486567 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:09.486778 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:09.492328 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:09.493875 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:32:09.494728 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:32:09.496416 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:09.501453 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:09.521106 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:09.562366 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:32:09.749938 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:09.767392 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:09.772233 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:09.772747 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:09.773689 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:09.788398 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:09.844604 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:32:09.893324 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:32:09.934496 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:32:09.938493 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:32:09.939059 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:09.947651 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 14:32:09.949224 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:32:09.959796 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:09.985795 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:09.989562 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:09.994189 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:10.000634 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:10.015113 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:10.035849 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:10.086895 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:32:10.143247 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:32:10.275018 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:10.276085 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:10.284582 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:10.287844 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:10.304808 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:10.318353 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:10.330881 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:32:10.339582 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:32:10.354646 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:10.484797 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:32:10.509446 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:10.544467 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:10.592342 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:10.597079 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:10.611733 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:10.619459 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:10.637239 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:10.666050 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:10.672372 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:10.697997 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:10.700063 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:10.701280 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:10.776623 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:10.818940 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:32:10.867066 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:32:10.931865 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:32:10.943014 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:10.952762 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:10.976130 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:10.988974 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:10.993935 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:11.000396 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:11.011096 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:11.104914 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:11.183598 19032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 14:32:11.184834 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:32:11.239622 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:11.244479 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:32:11.248541 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:11.279865 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:11.288755 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:11.302573 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:11.311769 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:11.317435 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:11.326685 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:11.353305 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:11.388985 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:11.392052 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:11.395484 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:11.437729 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:11.466041 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:32:11.516364 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:32:11.565543 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:11.618461 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:32:11.622457 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:11.623214 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:11.644865 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:32:11.649485 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:11.651098 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:11.662456 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:11.669775 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:11.810807 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:11.815544 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:32:11.840438 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:32:11.860648 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:32:11.873010 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:32:11.891409 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:11.900339 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:11.918776 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:11.924252 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:11.938190 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:11.942036 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:11.960222 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:32:12.019819 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:32:12.035511 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:12.054013 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:32:12.152938 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:32:12.180629 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:32:12.213649 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:12.232414 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:32:12.239179 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:12.249198 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:12.262851 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:12.269382 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:12.273603 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:12.280524 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:12.288676 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:12.306702 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:12.316507 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:12.331523 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:12.338301 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:12.373909 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:12.445261 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:32:12.449429 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:32:12.527952 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:32:12.536128 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:32:12.625665 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:12.632282 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:12.637003 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:12.637920 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:32:12.651615 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:12.656323 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:32:12.657384 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:12.686501 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:12.689132 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:32:12.708128 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:12.714107 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:12.732106 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:12.732393 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:32:12.768610 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:12.788872 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:12.794044 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:32:12.808431 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:12.809598 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:12.840565 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:32:12.844174 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:12.993508 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240502 14:32:13.029378 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:13.190680 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:13.271636 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:13.293833 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:32:13.295194 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:13.306799 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:32:13.312713 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:13.336108 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:13.383075 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:13.403043 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:13.451331 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:13.463250 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:13.543464 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:13.546311 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 77 similar messages]
I20240502 14:32:13.550209 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:13.558818 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:13.572575 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:13.579303 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:13.605262 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:32:13.629505 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:13.679662 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:32:13.757371 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:32:13.788411 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:13.801584 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:13.817147 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:13.825397 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:13.825795 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:13.826974 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:13.849455 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:32:13.978018 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:32:13.992897 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:32:14.019994 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:14.022142 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:14.034026 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:14.042095 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:14.055199 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:14.066320 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:14.102195 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:14.192101 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:32:14.193049 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:14.204644 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:32:14.262629 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:32:14.290896 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:14.312006 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:14.314314 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:14.315932 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:14.353790 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:14.354962 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:14.387058 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:32:14.443383 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:32:14.467453 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:32:14.480755 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:32:14.503180 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:14.523176 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:32:14.528609 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:14.546049 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:32:14.551896 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:14.555676 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:14.561527 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:14.589970 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:14.591740 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:14.616664 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:32:14.625098 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:32:14.727322 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:32:14.788601 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:32:14.791967 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:14.798519 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:14.810811 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:14.848084 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:14.858120 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:14.859265 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:32:14.870285 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:14.878154 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:32:14.922852 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:14.931319 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:14.957981 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240502 14:32:14.978984 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:14.985908 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:15.023164 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:32:15.043895 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:15.068686 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:15.085217 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:15.086337 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:15.168099 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:15.197472 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:32:15.199623 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:32:15.203238 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:32:15.270001 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:15.279495 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:15.291725 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:15.298234 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:15.321873 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:15.348469 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:15.350123 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:32:15.451581 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:32:15.501421 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:15.588233 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:15.601554 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:15.611083 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:15.621102 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:32:15.633260 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:32:15.635506 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:15.640175 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:15.663951 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:15.668853 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:15.760439 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:15.769920 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:15.828086 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:15.832720 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:32:15.868505 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:15.870018 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:15.886691 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:15.952371 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:16.048983 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:32:16.076185 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:16.078816 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:16.091514 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:16.093598 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:16.094719 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:16.101488 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:16.157634 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:32:16.277247 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:16.322100 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:32:16.332470 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:16.336192 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240502 14:32:16.337414 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:16.342939 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:16.349056 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:16.358475 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:16.359133 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:16.446014 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:32:16.525122 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:16.554639 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:16.590276 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:16.592164 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:16.614894 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:16.634162 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:16.634606 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:16.648823 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:32:16.663035 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:32:16.771026 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:16.775952 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:32:16.777058 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:16.786409 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:16.796742 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:16.799124 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:16.812520 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:32:16.828969 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:16.859113 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:32:16.896538 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:17.019757 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:32:17.030251 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:17.054010 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:17.099184 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:17.102495 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:17.102903 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:17.105454 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:17.105486 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:32:17.143323 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:32:17.163758 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:17.168587 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:32:17.176143 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:32:17.192557 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:32:17.194891 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:32:17.248462 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:17.285686 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:17.292208 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:17.293089 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:17.302430 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:17.305450 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:17.359037 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:32:17.394788 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:32:17.438972 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:32:17.478546 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:32:17.496450 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:17.597486 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:17.608731 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:17.624045 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:17.631819 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:17.646636 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:17.692627 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:17.698657 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:32:17.702616 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:32:17.710253 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:32:17.717242 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:17.799347 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:32:17.867921 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:32:17.928797 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:17.933094 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:17.949617 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:32:17.954138 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:17.966626 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:32:17.987314 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:17.993899 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:17.995087 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:32:17.999477 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:18.000276 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:18.117149 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:18.134655 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:18.143474 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:18.151352 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:18.157954 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:18.160555 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:18.186249 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:18.226270 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 14:32:18.311120 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:32:18.362309 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:32:18.371624 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:18.380667 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:18.384932 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:18.390311 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:18.394114 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:18.396162 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:18.487341 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:32:18.521427 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:18.538159 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:18.557646 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240502 14:32:18.701965 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:18.720353 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:18.735256 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:18.757303 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:18.770032 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:18.777877 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:18.781077 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:18.796819 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:18.870765 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:32:18.876255 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:18.902359 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:18.926326 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:18.931221 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:19.044679 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:32:19.072376 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:19.080116 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:19.083643 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:19.085103 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:32:19.104532 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:19.107270 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:19.108412 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:19.110589 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:32:19.138873 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:19.146320 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:32:19.154773 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:19.197171 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:32:19.249377 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:32:19.337910 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:32:19.359814 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:19.360029 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:19.378368 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:32:19.378464 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:32:19.396034 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:19.404297 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:19.405669 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:19.417819 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:19.566179 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:19.590641 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:19.596724 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:32:19.616045 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:19.629885 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:19.654816 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:32:19.655913 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:19.666896 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:19.669435 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:19.680137 20560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 14:32:19.681814 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:19.682389 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:32:19.720533 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:32:19.731377 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:32:19.763223 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:19.767225 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:32:19.792968 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:19.793068 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:19.803596 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:19.809260 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:19.814733 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:19.954600 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:32:19.985713 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 14:32:20.001050 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:32:20.026276 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:20.030723 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:20.054052 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:20.054401 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:20.054813 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:20.072010 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:20.075215 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:20.113029 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:32:20.151753 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:20.196185 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:32:20.215394 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:20.222265 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:20.226408 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:20.252482 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:20.254623 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:20.257228 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:20.293141 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:32:20.373201 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:32:20.418417 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:32:20.436463 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:32:20.464854 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:20.466187 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:32:20.482295 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:20.484695 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:20.492591 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:20.493319 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:20.516182 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:20.525523 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:20.662698 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:32:20.736111 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:20.748888 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:20.756747 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:20.788975 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:20.789146 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:20.800375 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:20.848950 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:32:20.862733 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:20.908346 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:20.926301 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:20.955828 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:20.963801 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:32:21.002504 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:21.002983 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:21.010390 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:21.032559 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:21.206756 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:21.217748 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:21.232815 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:21.236327 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:21.241124 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:21.264719 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:21.268309 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:21.496260 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:32:21.509932 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:21.541003 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:21.541574 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:21.552199 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:21.562052 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:21.569342 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:32:21.569871 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:21.588999 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 14:32:21.591445 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:21.624047 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:21.635535 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:32:21.679724 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:21.682421 20561 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.72:35871, user_credentials={real_user=slave}} blocked reactor thread for 116361us
W20240502 14:32:21.725881 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:32:21.726487 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:21.738879 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:21.747323 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:32:21.751437 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:21.756022 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:21.802558 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:32:21.807911 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:32:21.822826 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:32:21.848227 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:32:21.891625 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:21.951002 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:21.954910 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:21.966274 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:21.968621 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:21.975982 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:21.981962 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:22.018666 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:22.024003 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:32:22.061532 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:32:22.084648 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:32:22.164042 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:32:22.165095 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:32:22.173133 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:22.177358 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:32:22.188228 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:22.191887 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:32:22.246542 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:22.248142 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:22.250641 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:22.252177 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:22.270135 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:32:22.274120 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:32:22.466820 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:22.481840 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:22.491672 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:22.499711 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:22.510998 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:22.552040 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:22.568693 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:22.575107 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:22.644222 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:32:22.661386 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:22.688766 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:22.690294 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:22.707635 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:22.724925 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:22.726007 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:22.780257 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:32:22.780478 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:32:22.827955 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:32:22.893034 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:32:22.924604 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:32:22.929708 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:32:22.950098 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:22.953969 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:32:22.960781 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:22.967204 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:22.972652 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:22.974218 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:22.977267 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:22.986979 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:32:22.992486 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:23.046842 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:23.172564 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:23.179304 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:23.201886 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:23.211091 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:23.218526 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:23.228063 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:23.229920 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 14:32:23.331485 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:32:23.425419 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:23.479817 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:23.486375 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:23.486745 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:23.489601 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:23.494405 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:23.519193 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:23.556416 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:32:23.633566 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 14:32:23.651665 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:23.750516 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:32:23.750627 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:23.772666 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:23.832093 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:23.863360 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:23.876493 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:23.880177 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:23.885980 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:23.894290 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:23.895787 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:23.910840 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:23.930223 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:23.942854 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:32:23.943583 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:23.958305 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:24.032094 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:32:24.054116 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:24.151350 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:24.164955 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:24.166456 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:24.195611 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:32:24.200311 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:24.200389 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:24.203492 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:24.285898 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:32:24.320884 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:32:24.376821 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:32:24.388106 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:32:24.430243 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:24.441641 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:24.443841 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:24.445765 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:24.445868 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:24.462580 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:32:24.479712 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:24.520191 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:32:24.559568 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:32:24.565389 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:24.576202 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:32:24.624101 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:32:24.640208 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:24.660104 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:24.662521 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:24.688117 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:24.707692 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:24.709992 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:24.713287 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:32:24.728958 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:24.748199 18769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.68:41463, user_credentials={real_user=slave}} blocked reactor thread for 48932.4us
W20240502 14:32:24.756595 18768 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.72:35871, user_credentials={real_user=slave}} blocked reactor thread for 56044.8us
W20240502 14:32:24.787123 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:32:24.854287 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:32:24.885098 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:32:24.921094 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:24.927743 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:24.934994 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:24.942576 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:24.943625 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:24.947577 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:24.974252 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:32:24.998085 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:25.076987 20562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240502 14:32:25.090729 18635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 14:32:25.131804 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:32:25.200464 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:25.202546 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:25.219476 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:25.222558 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:25.227371 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:25.233716 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:32:25.234417 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:25.235239 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:25.235241 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:32:25.238915 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:32:25.263176 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:32:25.317791 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:32:25.454438 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:32:25.487614 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:32:25.493743 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:25.512249 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:25.519810 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:25.522401 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:25.537524 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:25.556149 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:25.558470 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:25.586340 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:25.607681 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:25.619062 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:25.623926 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:25.646059 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:25.647042 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:25.656073 18637 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.67:45409, user_credentials={real_user=slave}} blocked reactor thread for 87942.7us
I20240502 14:32:25.669360 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:25.781641 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:32:25.924871 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:32:25.942430 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:25.946560 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:25.952978 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:25.961735 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:25.964569 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:25.965102 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:26.015898 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:26.094074 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:26.126770 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:26.127173 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:26.145807 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:26.149281 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:26.161092 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:26.172225 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:26.309543 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:26.336056 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:32:26.432780 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:26.445451 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:26.467151 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:26.471843 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:26.509572 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:26.510006 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:32:26.510556 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:26.642985 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:32:26.643482 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:26.684574 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:32:26.714123 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:32:26.721050 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:26.732955 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:32:26.741259 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:26.768957 19030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 14:32:26.771454 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:26.813889 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:32:26.838924 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:32:26.844301 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:26.844671 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:26.871058 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:26.871510 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:26.903319 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:26.908695 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:26.943289 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:32:26.948930 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:26.960347 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:26.975901 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:26.989765 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:26.999365 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:32:27.005896 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:32:27.012501 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:32:27.058403 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:32:27.099009 18635 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:32:27.185681 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:27.192708 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:27.199216 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:27.215518 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:27.218029 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:27.229686 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:27.235301 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:27.249543 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:27.253270 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:32:27.314875 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:32:27.372879 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:32:27.387706 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:32:27.429775 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:32:27.499176 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:27.515931 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:27.516793 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:27.521445 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:27.527029 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:27.546006 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:27.568396 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:32:27.571378 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:32:27.609563 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:27.612295 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:27.612607 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:27.642272 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:27.665984 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:27.672629 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:27.685710 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:27.730319 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:27.735808 18898 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 14:32:27.880344 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:32:27.885700 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:32:27.910547 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:27.923208 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:27.924060 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:27.930799 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:27.933715 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:27.954340 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:28.007045 20964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:32:28.007894 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:28.094555 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:28.127231 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:28.146485 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:28.158120 18898 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 14:32:28.158252 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:28.171835 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:28.174228 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:28.174983 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:28.221632 18766 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:32:28.253547 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 14:32:28.368695 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:32:28.390772 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:28.393292 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:32:28.398672 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:28.428002 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:28.457028 18766 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:32:28.460434 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:28.467037 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:28.467141 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:28.505544 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:28.584400 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:28.597658 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:28.604449 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:28.617576 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:28.624213 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:28.625437 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:28.627884 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:28.732772 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240502 14:32:28.849898 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:28.875048 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:28.879914 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:28.886750 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:28.902459 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:28.903874 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:32:28.907516 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:28.932901 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:29.076892 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:29.099885 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:29.105082 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:29.117624 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:29.121335 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:29.127825 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:29.209319 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:29.224733 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:32:29.254240 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 14:32:29.279822 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:29.301461 18635 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:32:29.376781 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:32:29.376989 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:32:29.377781 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:29.382575 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:29.394582 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:29.413748 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:29.420575 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:29.438630 7748 raft_consensus.cc:998] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b: Attempting to remove follower a14c220ad2cc4bf9bd7dad65f4803eb0 from the Raft config. Reason: Leader has been unable to successfully communicate with peer a14c220ad2cc4bf9bd7dad65f4803eb0 for more than 300 seconds (300.071s)
I20240502 14:32:29.439608 7748 consensus_queue.cc:237] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:32:29.445731 18962 raft_consensus.cc:1261] T 96662a3d9e1b4f03b05464ddd06b93d6 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:32:29.446630 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:29.446583 7748 consensus_queue.cc:1035] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:32:29.453305 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:32:29.462181 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:32:29.469307 7748 raft_consensus.cc:2932] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:32:29.490378 18962 raft_consensus.cc:2932] T 96662a3d9e1b4f03b05464ddd06b93d6 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:32:29.491362 18560 catalog_manager.cc:5484] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b reported cstate change: config changed from index 6 to 7, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:32:29.498602 7795 raft_consensus.cc:998] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4: Attempting to remove follower a14c220ad2cc4bf9bd7dad65f4803eb0 from the Raft config. Reason: Leader has been unable to successfully communicate with peer a14c220ad2cc4bf9bd7dad65f4803eb0 for more than 300 seconds (300.111s)
I20240502 14:32:29.499307 7795 consensus_queue.cc:237] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
W20240502 14:32:29.503868 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:32:29.510959 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:32:29.519577 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:32:29.521225 18962 raft_consensus.cc:1261] T 67827c0262724897a05bf80df579bbf3 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:32:29.521840 7962 consensus_queue.cc:1035] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:32:29.529026 19030 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:32:29.530023 7795 raft_consensus.cc:998] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4: Attempting to remove follower a14c220ad2cc4bf9bd7dad65f4803eb0 from the Raft config. Reason: Leader has been unable to successfully communicate with peer a14c220ad2cc4bf9bd7dad65f4803eb0 for more than 300 seconds (300.179s)
I20240502 14:32:29.530781 7795 consensus_queue.cc:237] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
W20240502 14:32:29.532027 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:32:29.542826 19093 raft_consensus.cc:1261] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:32:29.543438 8040 consensus_queue.cc:1035] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:32:29.544312 7962 raft_consensus.cc:2932] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:32:29.553773 18555 catalog_manager.cc:5484] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index 6 to 7, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
W20240502 14:32:29.555368 18544 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:29.557581 18963 raft_consensus.cc:2932] T 67827c0262724897a05bf80df579bbf3 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:32:29.575106 8040 raft_consensus.cc:2932] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
W20240502 14:32:29.579077 18544 catalog_manager.cc:4573] TS a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 96662a3d9e1b4f03b05464ddd06b93d6: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:29.580860 19093 raft_consensus.cc:2932] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
W20240502 14:32:29.584285 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:32:29.585529 18555 catalog_manager.cc:5484] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index 5 to 6, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:32:29.598019 18833 consensus_queue.cc:237] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:32:29.599999 19093 raft_consensus.cc:1261] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:32:29.601445 8040 consensus_queue.cc:1035] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:32:29.601723 18769 consensus_peers.cc:485] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409): Couldn't send request to peer c902805060df4df9b5599d0e94b2dc06. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:32:29.604384 7795 raft_consensus.cc:2932] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
I20240502 14:32:29.605062 19095 raft_consensus.cc:2932] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
I20240502 14:32:29.609401 18543 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 5be0d6f4f1874c79b906c4bdc34b0ce9 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:32:29.610841 18555 catalog_manager.cc:5484] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index 6 to 7, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) added. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
W20240502 14:32:29.668424 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:32:29.688154 7748 raft_consensus.cc:998] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b: Attempting to remove follower a14c220ad2cc4bf9bd7dad65f4803eb0 from the Raft config. Reason: Leader has been unable to successfully communicate with peer a14c220ad2cc4bf9bd7dad65f4803eb0 for more than 300 seconds (300.299s)
I20240502 14:32:29.688807 7748 consensus_queue.cc:237] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:32:29.703291 6611 raft_consensus.cc:998] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06: Attempting to remove follower a14c220ad2cc4bf9bd7dad65f4803eb0 from the Raft config. Reason: Leader has been unable to successfully communicate with peer a14c220ad2cc4bf9bd7dad65f4803eb0 for more than 300 seconds (300.254s)
I20240502 14:32:29.704191 6611 consensus_queue.cc:237] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
W20240502 14:32:29.719255 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:32:29.720360 19095 raft_consensus.cc:1261] T 246a8425400f4890a5264c4ae8a627c5 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:32:29.721104 8073 consensus_queue.cc:1035] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:32:29.727576 21029 raft_consensus.cc:1261] T b315327141954b53a0f87c2d1dc14b9b P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 14:32:29.727839 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:32:29.728375 7748 consensus_queue.cc:1035] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:32:29.732007 8073 raft_consensus.cc:2932] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:32:29.734725 7748 raft_consensus.cc:2932] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:32:29.739614 19095 raft_consensus.cc:2932] T 246a8425400f4890a5264c4ae8a627c5 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:32:29.749437 6611 raft_consensus.cc:998] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06: Attempting to remove follower a14c220ad2cc4bf9bd7dad65f4803eb0 from the Raft config. Reason: Leader has been unable to successfully communicate with peer a14c220ad2cc4bf9bd7dad65f4803eb0 for more than 300 seconds (300.209s)
I20240502 14:32:29.750178 6611 consensus_queue.cc:237] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:32:29.749939 18560 catalog_manager.cc:5484] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 reported cstate change: config changed from index 6 to 7, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:32:29.749959 21029 raft_consensus.cc:2932] T b315327141954b53a0f87c2d1dc14b9b P ff6acba046404ef0a6f41ad5e09d852f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:32:29.759809 18560 catalog_manager.cc:5484] T b315327141954b53a0f87c2d1dc14b9b P ff6acba046404ef0a6f41ad5e09d852f reported cstate change: config changed from index 8 to 9, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } } }
I20240502 14:32:29.761381 20625 raft_consensus.cc:1261] T 21af917743444f2db932636a78f33512 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:32:29.762035 6611 consensus_queue.cc:1035] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:32:29.763895 7795 raft_consensus.cc:998] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4: Attempting to remove follower a14c220ad2cc4bf9bd7dad65f4803eb0 from the Raft config. Reason: Leader has been unable to successfully communicate with peer a14c220ad2cc4bf9bd7dad65f4803eb0 for more than 300 seconds (300.146s)
I20240502 14:32:29.764559 7795 consensus_queue.cc:237] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
W20240502 14:32:29.772899 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:29.776949 6611 raft_consensus.cc:2932] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:32:29.778570 20625 raft_consensus.cc:2932] T 21af917743444f2db932636a78f33512 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:32:29.779492 19093 raft_consensus.cc:1261] T 6acde0e5509340fbb52bca18645a77b7 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:32:29.781864 7795 consensus_queue.cc:1035] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:32:29.794577 7962 raft_consensus.cc:2932] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:32:29.795342 19093 raft_consensus.cc:2932] T 6acde0e5509340fbb52bca18645a77b7 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:32:29.804059 18553 catalog_manager.cc:5484] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 reported cstate change: config changed from index 7 to 8, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:32:29.806075 18560 catalog_manager.cc:5484] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index 5 to 6, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
W20240502 14:32:29.825053 19030 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:32:29.825562 18963 consensus_queue.cc:237] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:32:29.826346 5851 raft_consensus.cc:998] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8: Attempting to remove follower a14c220ad2cc4bf9bd7dad65f4803eb0 from the Raft config. Reason: Leader has been unable to successfully communicate with peer a14c220ad2cc4bf9bd7dad65f4803eb0 for more than 300 seconds (300.066s)
I20240502 14:32:29.827785 5851 consensus_queue.cc:237] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:32:29.830384 18833 consensus_queue.cc:237] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:32:29.841054 18701 raft_consensus.cc:1261] T bad03a4ec8494938839d7d6eef8c684e P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 14:32:29.841598 18766 consensus_peers.cc:485] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749): Couldn't send request to peer 2db304b538e14857b7b1cc272d0db94b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6acde0e5509340fbb52bca18645a77b7. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:32:29.841646 5851 consensus_queue.cc:1035] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 14:32:29.844839 18900 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463): Couldn't send request to peer 007b2e79dc424a6f80141180f072f2d8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:32:29.847268 19030 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 14:32:29.847972 8106 raft_consensus.cc:998] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8: Attempting to remove follower a14c220ad2cc4bf9bd7dad65f4803eb0 from the Raft config. Reason: Leader has been unable to successfully communicate with peer a14c220ad2cc4bf9bd7dad65f4803eb0 for more than 300 seconds (300.140s)
I20240502 14:32:29.848799 8106 consensus_queue.cc:237] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:32:29.851392 20626 raft_consensus.cc:1261] T 21af917743444f2db932636a78f33512 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:32:29.852435 6611 consensus_queue.cc:1035] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:32:29.857921 18699 raft_consensus.cc:1261] T e5fb3b049ba94af998791640bd0e31c5 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:32:29.858541 5851 consensus_queue.cc:1035] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:32:29.859584 20626 raft_consensus.cc:2932] T 21af917743444f2db932636a78f33512 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:32:29.867225 18553 catalog_manager.cc:5484] T 21af917743444f2db932636a78f33512 P a834edffbcb2469aae598b069aab0681 reported cstate change: config changed from index 8 to 9, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) added. New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:32:29.868355 8073 raft_consensus.cc:2932] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:32:29.876075 8106 raft_consensus.cc:2932] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:32:29.877043 18699 raft_consensus.cc:2932] T e5fb3b049ba94af998791640bd0e31c5 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:32:29.879015 18546 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 21af917743444f2db932636a78f33512 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:32:29.887877 19097 raft_consensus.cc:1261] T 6acde0e5509340fbb52bca18645a77b7 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:32:29.888867 7962 consensus_queue.cc:1035] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:32:29.891270 8107 raft_consensus.cc:998] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681: Attempting to remove follower a14c220ad2cc4bf9bd7dad65f4803eb0 from the Raft config. Reason: Leader has been unable to successfully communicate with peer a14c220ad2cc4bf9bd7dad65f4803eb0 for more than 300 seconds (300.169s)
I20240502 14:32:29.891880 18560 catalog_manager.cc:5484] T e5fb3b049ba94af998791640bd0e31c5 P 2db304b538e14857b7b1cc272d0db94b reported cstate change: config changed from index 5 to 6, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:32:29.892361 7962 raft_consensus.cc:2932] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:32:29.896185 19097 raft_consensus.cc:2932] T 6acde0e5509340fbb52bca18645a77b7 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
W20240502 14:32:29.903410 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:32:29.908386 18543 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 6acde0e5509340fbb52bca18645a77b7 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:32:29.909004 18701 raft_consensus.cc:2932] T bad03a4ec8494938839d7d6eef8c684e P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:32:29.911139 19093 consensus_queue.cc:237] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:32:29.919237 8107 consensus_queue.cc:237] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:32:29.920020 18699 raft_consensus.cc:1261] T e5fb3b049ba94af998791640bd0e31c5 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 14:32:29.922909 19029 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009): Couldn't send request to peer fec04c50dc5b49878ffa3c3873df32e4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:32:29.933560 8102 raft_consensus.cc:2932] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:32:29.937777 8103 consensus_queue.cc:1035] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:32:29.942653 18555 catalog_manager.cc:5484] T bad03a4ec8494938839d7d6eef8c684e P 2db304b538e14857b7b1cc272d0db94b reported cstate change: config changed from index 5 to 6, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:32:29.943542 21029 raft_consensus.cc:1261] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f [term 3 FOLLOWER]: Refusing update from remote peer a834edffbcb2469aae598b069aab0681: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240502 14:32:29.943907 18560 catalog_manager.cc:5484] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index 6 to 7, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) added. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:32:29.945116 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:29.950538 8107 consensus_queue.cc:1035] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240502 14:32:29.958384 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:29.970003 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:29.974169 19095 consensus_queue.cc:237] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:32:29.975791 8105 raft_consensus.cc:2932] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 11 to 12, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:32:29.976439 21028 raft_consensus.cc:2932] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 11 to 12, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:32:29.984210 18553 catalog_manager.cc:5484] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f reported cstate change: config changed from index 11 to 12, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New cstate: current_term: 3 leader_uuid: "a834edffbcb2469aae598b069aab0681" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:32:29.987905 18701 raft_consensus.cc:1261] T bad03a4ec8494938839d7d6eef8c684e P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:32:29.987854 8102 raft_consensus.cc:2932] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } }
I20240502 14:32:29.994185 18701 raft_consensus.cc:2932] T e5fb3b049ba94af998791640bd0e31c5 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } }
I20240502 14:32:29.995036 18546 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet e5fb3b049ba94af998791640bd0e31c5 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:32:29.996181 8102 consensus_queue.cc:1035] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:32:29.999589 18560 catalog_manager.cc:5484] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: config changed from index 6 to 7, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) added. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } }
I20240502 14:32:30.005584 8102 raft_consensus.cc:2932] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
W20240502 14:32:30.007018 19032 consensus_peers.cc:485] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 -> Peer c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409): Couldn't send request to peer c902805060df4df9b5599d0e94b2dc06. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:32:30.007468 18701 raft_consensus.cc:2932] T bad03a4ec8494938839d7d6eef8c684e P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
I20240502 14:32:30.022544 18546 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet bad03a4ec8494938839d7d6eef8c684e with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:32:30.025671 18553 catalog_manager.cc:5484] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: config changed from index 6 to 7, VOTER c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) added. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
I20240502 14:32:30.030045 20626 consensus_queue.cc:237] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:32:30.033080 21028 raft_consensus.cc:1261] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f [term 3 FOLLOWER]: Refusing update from remote peer a834edffbcb2469aae598b069aab0681: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 14:32:30.033918 8107 consensus_queue.cc:1035] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 14:32:30.037690 8107 raft_consensus.cc:2932] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:32:30.041901 21028 raft_consensus.cc:2932] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
W20240502 14:32:30.046618 18766 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.5.107.70:46021, user_credentials={real_user=slave}} blocked reactor thread for 88229.9us
I20240502 14:32:30.056818 18546 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 522014370f0a4cc3bd31aa00d6a05c5e with cas_config_opid_index 12: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:32:30.058044 18553 catalog_manager.cc:5484] T 522014370f0a4cc3bd31aa00d6a05c5e P ff6acba046404ef0a6f41ad5e09d852f reported cstate change: config changed from index 12 to 13, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) added. New cstate: current_term: 3 leader_uuid: "a834edffbcb2469aae598b069aab0681" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
W20240502 14:32:30.094178 20560 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749): Couldn't send request to peer 2db304b538e14857b7b1cc272d0db94b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:32:30.096747 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:30.102427 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:30.112169 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:30.117303 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:30.164057 8142 ts_tablet_manager.cc:902] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06: Initiating tablet copy from peer fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
W20240502 14:32:30.173322 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 14:32:30.183691 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:30.191191 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:30.215513 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:30.257695 8142 tablet_copy_client.cc:308] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:33009
I20240502 14:32:30.263448 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:30.269415 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:30.286855 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:32:30.303872 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:32:30.304000 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:30.306385 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:30.357161 8109 raft_consensus.cc:998] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06: Attempting to remove follower a14c220ad2cc4bf9bd7dad65f4803eb0 from the Raft config. Reason: Leader has been unable to successfully communicate with peer a14c220ad2cc4bf9bd7dad65f4803eb0 for more than 300 seconds (300.599s)
I20240502 14:32:30.357865 8109 consensus_queue.cc:237] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
W20240502 14:32:30.361750 20964 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.5.107.71:39195, user_credentials={real_user=slave}} blocked reactor thread for 66082.4us
W20240502 14:32:30.366353 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:30.373634 8153 ts_tablet_manager.cc:902] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8: Initiating tablet copy from peer c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:32:30.378062 21028 raft_consensus.cc:1261] T b15d11209e45473e863766e5cd6e822f P ff6acba046404ef0a6f41ad5e09d852f [term 2 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:32:30.378978 8088 consensus_queue.cc:1035] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:32:30.382601 8088 raft_consensus.cc:2932] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:32:30.387117 21029 raft_consensus.cc:2932] T b15d11209e45473e863766e5cd6e822f P ff6acba046404ef0a6f41ad5e09d852f [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:32:30.393015 18553 catalog_manager.cc:5484] T b15d11209e45473e863766e5cd6e822f P ff6acba046404ef0a6f41ad5e09d852f reported cstate change: config changed from index 8 to 9, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New cstate: current_term: 2 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:32:30.397922 8158 ts_tablet_manager.cc:902] T 6acde0e5509340fbb52bca18645a77b7 P 2db304b538e14857b7b1cc272d0db94b: Initiating tablet copy from peer fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
I20240502 14:32:30.399580 18963 consensus_queue.cc:237] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
W20240502 14:32:30.406826 20562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 14:32:30.414140 8153 tablet_copy_client.cc:308] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.67:45409
I20240502 14:32:30.417450 8158 tablet_copy_client.cc:308] T 6acde0e5509340fbb52bca18645a77b7 P 2db304b538e14857b7b1cc272d0db94b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:33009
I20240502 14:32:30.417834 18985 tablet_copy_service.cc:140] P c902805060df4df9b5599d0e94b2dc06: Received BeginTabletCopySession request for tablet 21af917743444f2db932636a78f33512 from peer 007b2e79dc424a6f80141180f072f2d8 ({username='slave'} at 127.5.107.68:34453)
I20240502 14:32:30.417999 18985 tablet_copy_service.cc:161] P c902805060df4df9b5599d0e94b2dc06: Beginning new tablet copy session on tablet 21af917743444f2db932636a78f33512 from peer 007b2e79dc424a6f80141180f072f2d8 at {username='slave'} at 127.5.107.68:34453: session id = 007b2e79dc424a6f80141180f072f2d8-21af917743444f2db932636a78f33512
I20240502 14:32:30.420454 8073 raft_consensus.cc:2932] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:32:30.421285 18854 tablet_copy_service.cc:140] P fec04c50dc5b49878ffa3c3873df32e4: Received BeginTabletCopySession request for tablet 5be0d6f4f1874c79b906c4bdc34b0ce9 from peer c902805060df4df9b5599d0e94b2dc06 ({username='slave'} at 127.5.107.67:50775)
I20240502 14:32:30.421435 18854 tablet_copy_service.cc:161] P fec04c50dc5b49878ffa3c3873df32e4: Beginning new tablet copy session on tablet 5be0d6f4f1874c79b906c4bdc34b0ce9 from peer c902805060df4df9b5599d0e94b2dc06 at {username='slave'} at 127.5.107.67:50775: session id = c902805060df4df9b5599d0e94b2dc06-5be0d6f4f1874c79b906c4bdc34b0ce9
I20240502 14:32:30.424338 18853 tablet_copy_service.cc:140] P fec04c50dc5b49878ffa3c3873df32e4: Received BeginTabletCopySession request for tablet 6acde0e5509340fbb52bca18645a77b7 from peer 2db304b538e14857b7b1cc272d0db94b ({username='slave'} at 127.5.107.65:33355)
I20240502 14:32:30.424507 18853 tablet_copy_service.cc:161] P fec04c50dc5b49878ffa3c3873df32e4: Beginning new tablet copy session on tablet 6acde0e5509340fbb52bca18645a77b7 from peer 2db304b538e14857b7b1cc272d0db94b at {username='slave'} at 127.5.107.65:33355: session id = 2db304b538e14857b7b1cc272d0db94b-6acde0e5509340fbb52bca18645a77b7
I20240502 14:32:30.424824 18854 tablet_copy_source_session.cc:205] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P fec04c50dc5b49878ffa3c3873df32e4: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:32:30.426810 18985 tablet_copy_source_session.cc:205] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:32:30.427675 18853 tablet_copy_source_session.cc:205] T 6acde0e5509340fbb52bca18645a77b7 P fec04c50dc5b49878ffa3c3873df32e4: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:32:30.429102 18546 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet b15d11209e45473e863766e5cd6e822f with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:32:30.429984 21028 raft_consensus.cc:2932] T b15d11209e45473e863766e5cd6e822f P ff6acba046404ef0a6f41ad5e09d852f [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:32:30.436178 8158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6acde0e5509340fbb52bca18645a77b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:32:30.437541 18553 catalog_manager.cc:5484] T b15d11209e45473e863766e5cd6e822f P ff6acba046404ef0a6f41ad5e09d852f reported cstate change: config changed from index 9 to 10, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) added. New cstate: current_term: 2 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:32:30.442139 8142 ts_tablet_manager.cc:1875] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:32:30.443385 8153 ts_tablet_manager.cc:1875] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8: Deleting tablet data with delete state TABLET_DATA_COPYING
W20240502 14:32:30.444444 18898 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749): Couldn't send request to peer 2db304b538e14857b7b1cc272d0db94b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:32:30.448593 8158 tablet_copy_client.cc:783] T 6acde0e5509340fbb52bca18645a77b7 P 2db304b538e14857b7b1cc272d0db94b: tablet copy: Starting download of 5 data blocks...
I20240502 14:32:30.451910 8142 ts_tablet_manager.cc:1888] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 14:32:30.452178 8142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be0d6f4f1874c79b906c4bdc34b0ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:32:30.453827 8153 ts_tablet_manager.cc:1888] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 14:32:30.454169 8153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21af917743444f2db932636a78f33512. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:32:30.456180 8142 tablet_copy_client.cc:783] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06: tablet copy: Starting download of 5 data blocks...
I20240502 14:32:30.460508 8158 tablet_copy_client.cc:647] T 6acde0e5509340fbb52bca18645a77b7 P 2db304b538e14857b7b1cc272d0db94b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:32:30.461658 8153 tablet_copy_client.cc:783] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8: tablet copy: Starting download of 5 data blocks...
I20240502 14:32:30.462754 8158 tablet_copy_client.cc:523] T 6acde0e5509340fbb52bca18645a77b7 P 2db304b538e14857b7b1cc272d0db94b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:32:30.466961 8158 tablet_bootstrap.cc:492] T 6acde0e5509340fbb52bca18645a77b7 P 2db304b538e14857b7b1cc272d0db94b: Bootstrap starting.
I20240502 14:32:30.468526 8142 tablet_copy_client.cc:647] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06: tablet copy: Starting download of 1 WAL segments...
I20240502 14:32:30.470937 8142 tablet_copy_client.cc:523] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:32:30.474064 8153 tablet_copy_client.cc:647] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8: tablet copy: Starting download of 1 WAL segments...
I20240502 14:32:30.474792 8142 tablet_bootstrap.cc:492] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06: Bootstrap starting.
I20240502 14:32:30.476429 8153 tablet_copy_client.cc:523] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:32:30.481199 8153 tablet_bootstrap.cc:492] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8: Bootstrap starting.
I20240502 14:32:30.502429 8158 tablet_bootstrap.cc:492] T 6acde0e5509340fbb52bca18645a77b7 P 2db304b538e14857b7b1cc272d0db94b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:32:30.503028 8158 tablet_bootstrap.cc:492] T 6acde0e5509340fbb52bca18645a77b7 P 2db304b538e14857b7b1cc272d0db94b: Bootstrap complete.
I20240502 14:32:30.503446 8158 ts_tablet_manager.cc:1367] T 6acde0e5509340fbb52bca18645a77b7 P 2db304b538e14857b7b1cc272d0db94b: Time spent bootstrapping tablet: real 0.037s user 0.022s sys 0.014s
I20240502 14:32:30.506201 8142 tablet_bootstrap.cc:492] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:32:30.506371 8158 raft_consensus.cc:348] T 6acde0e5509340fbb52bca18645a77b7 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:32:30.506659 8142 tablet_bootstrap.cc:492] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06: Bootstrap complete.
I20240502 14:32:30.506760 8158 raft_consensus.cc:729] T 6acde0e5509340fbb52bca18645a77b7 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Initialized, Role: FOLLOWER
I20240502 14:32:30.506978 8142 ts_tablet_manager.cc:1367] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06: Time spent bootstrapping tablet: real 0.032s user 0.027s sys 0.000s
I20240502 14:32:30.507167 8158 consensus_queue.cc:260] T 6acde0e5509340fbb52bca18645a77b7 P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:32:30.507777 8142 raft_consensus.cc:348] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:32:30.508010 8142 raft_consensus.cc:729] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Initialized, Role: FOLLOWER
I20240502 14:32:30.508322 8142 consensus_queue.cc:260] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:32:30.511469 8142 ts_tablet_manager.cc:1398] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 14:32:30.512434 18854 tablet_copy_service.cc:342] P fec04c50dc5b49878ffa3c3873df32e4: Request end of tablet copy session c902805060df4df9b5599d0e94b2dc06-5be0d6f4f1874c79b906c4bdc34b0ce9 received from {username='slave'} at 127.5.107.67:50775
I20240502 14:32:30.512557 18854 tablet_copy_service.cc:433] P fec04c50dc5b49878ffa3c3873df32e4: ending tablet copy session c902805060df4df9b5599d0e94b2dc06-5be0d6f4f1874c79b906c4bdc34b0ce9 on tablet 5be0d6f4f1874c79b906c4bdc34b0ce9 with peer c902805060df4df9b5599d0e94b2dc06
W20240502 14:32:30.515075 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:32:30.521785 8158 ts_tablet_manager.cc:1398] T 6acde0e5509340fbb52bca18645a77b7 P 2db304b538e14857b7b1cc272d0db94b: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240502 14:32:30.523975 18854 tablet_copy_service.cc:342] P fec04c50dc5b49878ffa3c3873df32e4: Request end of tablet copy session 2db304b538e14857b7b1cc272d0db94b-6acde0e5509340fbb52bca18645a77b7 received from {username='slave'} at 127.5.107.65:33355
I20240502 14:32:30.524109 18854 tablet_copy_service.cc:433] P fec04c50dc5b49878ffa3c3873df32e4: ending tablet copy session 2db304b538e14857b7b1cc272d0db94b-6acde0e5509340fbb52bca18645a77b7 on tablet 6acde0e5509340fbb52bca18645a77b7 with peer 2db304b538e14857b7b1cc272d0db94b
I20240502 14:32:30.528065 8158 ts_tablet_manager.cc:902] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b: Initiating tablet copy from peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119)
I20240502 14:32:30.537567 8172 ts_tablet_manager.cc:902] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4: Initiating tablet copy from peer 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:32:30.541074 18963 raft_consensus.cc:1203] T 5be0d6f4f1874c79b906c4bdc34b0ce9 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:32:30.546353 8142 ts_tablet_manager.cc:902] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06: Initiating tablet copy from peer 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
I20240502 14:32:30.574999 8172 tablet_copy_client.cc:308] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.68:41463
I20240502 14:32:30.576071 19117 tablet_copy_service.cc:140] P 007b2e79dc424a6f80141180f072f2d8: Received BeginTabletCopySession request for tablet e5fb3b049ba94af998791640bd0e31c5 from peer fec04c50dc5b49878ffa3c3873df32e4 ({username='slave'} at 127.5.107.66:33187)
I20240502 14:32:30.576243 19117 tablet_copy_service.cc:161] P 007b2e79dc424a6f80141180f072f2d8: Beginning new tablet copy session on tablet e5fb3b049ba94af998791640bd0e31c5 from peer fec04c50dc5b49878ffa3c3873df32e4 at {username='slave'} at 127.5.107.66:33187: session id = fec04c50dc5b49878ffa3c3873df32e4-e5fb3b049ba94af998791640bd0e31c5
I20240502 14:32:30.577658 8142 tablet_copy_client.cc:308] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.68:41463
I20240502 14:32:30.578323 19116 tablet_copy_service.cc:140] P 007b2e79dc424a6f80141180f072f2d8: Received BeginTabletCopySession request for tablet bad03a4ec8494938839d7d6eef8c684e from peer c902805060df4df9b5599d0e94b2dc06 ({username='slave'} at 127.5.107.67:34109)
I20240502 14:32:30.578500 19116 tablet_copy_service.cc:161] P 007b2e79dc424a6f80141180f072f2d8: Beginning new tablet copy session on tablet bad03a4ec8494938839d7d6eef8c684e from peer c902805060df4df9b5599d0e94b2dc06 at {username='slave'} at 127.5.107.67:34109: session id = c902805060df4df9b5599d0e94b2dc06-bad03a4ec8494938839d7d6eef8c684e
I20240502 14:32:30.582072 19117 tablet_copy_source_session.cc:205] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:32:30.582543 19116 tablet_copy_source_session.cc:205] T bad03a4ec8494938839d7d6eef8c684e P 007b2e79dc424a6f80141180f072f2d8: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:32:30.584381 8142 ts_tablet_manager.cc:1875] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:32:30.587723 8172 ts_tablet_manager.cc:1875] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:32:30.597877 8158 tablet_copy_client.cc:308] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.69:41119
I20240502 14:32:30.598644 20647 tablet_copy_service.cc:140] P a834edffbcb2469aae598b069aab0681: Received BeginTabletCopySession request for tablet 522014370f0a4cc3bd31aa00d6a05c5e from peer 2db304b538e14857b7b1cc272d0db94b ({username='slave'} at 127.5.107.65:53335)
I20240502 14:32:30.598798 20647 tablet_copy_service.cc:161] P a834edffbcb2469aae598b069aab0681: Beginning new tablet copy session on tablet 522014370f0a4cc3bd31aa00d6a05c5e from peer 2db304b538e14857b7b1cc272d0db94b at {username='slave'} at 127.5.107.65:53335: session id = 2db304b538e14857b7b1cc272d0db94b-522014370f0a4cc3bd31aa00d6a05c5e
I20240502 14:32:30.599076 8142 ts_tablet_manager.cc:1888] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 14:32:30.599324 8142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bad03a4ec8494938839d7d6eef8c684e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:32:30.601894 8153 tablet_bootstrap.cc:492] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:32:30.602263 8153 tablet_bootstrap.cc:492] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8: Bootstrap complete.
I20240502 14:32:30.602525 8153 ts_tablet_manager.cc:1367] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8: Time spent bootstrapping tablet: real 0.121s user 0.029s sys 0.008s
I20240502 14:32:30.602627 20647 tablet_copy_source_session.cc:205] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:32:30.603924 8158 ts_tablet_manager.cc:1875] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:32:30.604455 8172 ts_tablet_manager.cc:1888] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 14:32:30.604754 8172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5fb3b049ba94af998791640bd0e31c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:32:30.605937 19012 maintenance_manager.cc:419] P c902805060df4df9b5599d0e94b2dc06: Scheduling UndoDeltaBlockGCOp(5be0d6f4f1874c79b906c4bdc34b0ce9): 4586 bytes on disk
I20240502 14:32:30.606582 18901 maintenance_manager.cc:643] P c902805060df4df9b5599d0e94b2dc06: UndoDeltaBlockGCOp(5be0d6f4f1874c79b906c4bdc34b0ce9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240502 14:32:30.608071 8142 tablet_copy_client.cc:783] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06: tablet copy: Starting download of 5 data blocks...
I20240502 14:32:30.613911 8142 tablet_copy_client.cc:647] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06: tablet copy: Starting download of 1 WAL segments...
I20240502 14:32:30.617770 8142 tablet_copy_client.cc:523] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:32:30.618557 8153 raft_consensus.cc:348] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:32:30.619138 8153 raft_consensus.cc:729] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Initialized, Role: FOLLOWER
I20240502 14:32:30.619381 8172 tablet_copy_client.cc:783] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4: tablet copy: Starting download of 5 data blocks...
I20240502 14:32:30.619371 8153 consensus_queue.cc:260] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:32:30.621322 8153 ts_tablet_manager.cc:1398] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8: Time spent starting tablet: real 0.019s user 0.003s sys 0.000s
I20240502 14:32:30.622149 18985 tablet_copy_service.cc:342] P c902805060df4df9b5599d0e94b2dc06: Request end of tablet copy session 007b2e79dc424a6f80141180f072f2d8-21af917743444f2db932636a78f33512 received from {username='slave'} at 127.5.107.68:34453
I20240502 14:32:30.622292 18985 tablet_copy_service.cc:433] P c902805060df4df9b5599d0e94b2dc06: ending tablet copy session 007b2e79dc424a6f80141180f072f2d8-21af917743444f2db932636a78f33512 on tablet 21af917743444f2db932636a78f33512 with peer 007b2e79dc424a6f80141180f072f2d8
I20240502 14:32:30.625767 8142 tablet_bootstrap.cc:492] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06: Bootstrap starting.
I20240502 14:32:30.627449 8158 ts_tablet_manager.cc:1888] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 3.9
I20240502 14:32:30.627763 8158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 522014370f0a4cc3bd31aa00d6a05c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:32:30.627871 8172 tablet_copy_client.cc:647] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4: tablet copy: Starting download of 1 WAL segments...
I20240502 14:32:30.629508 8172 tablet_copy_client.cc:523] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:32:30.633584 8172 tablet_bootstrap.cc:492] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4: Bootstrap starting.
W20240502 14:32:30.635339 18544 catalog_manager.cc:4573] TS a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet bad03a4ec8494938839d7d6eef8c684e: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:30.642637 8158 tablet_copy_client.cc:783] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b: tablet copy: Starting download of 5 data blocks...
I20240502 14:32:30.653342 8158 tablet_copy_client.cc:647] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:32:30.655476 8158 tablet_copy_client.cc:523] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:32:30.659473 8158 tablet_bootstrap.cc:492] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b: Bootstrap starting.
I20240502 14:32:30.683919 8142 tablet_bootstrap.cc:492] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:32:30.684293 8142 tablet_bootstrap.cc:492] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06: Bootstrap complete.
I20240502 14:32:30.684525 8142 ts_tablet_manager.cc:1367] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06: Time spent bootstrapping tablet: real 0.060s user 0.013s sys 0.005s
I20240502 14:32:30.685350 8142 raft_consensus.cc:348] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:32:30.685580 8142 raft_consensus.cc:729] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Initialized, Role: FOLLOWER
I20240502 14:32:30.685829 8142 consensus_queue.cc:260] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:32:30.686396 8142 ts_tablet_manager.cc:1398] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 14:32:30.687114 19116 tablet_copy_service.cc:342] P 007b2e79dc424a6f80141180f072f2d8: Request end of tablet copy session c902805060df4df9b5599d0e94b2dc06-bad03a4ec8494938839d7d6eef8c684e received from {username='slave'} at 127.5.107.67:34109
I20240502 14:32:30.687264 19116 tablet_copy_service.cc:433] P 007b2e79dc424a6f80141180f072f2d8: ending tablet copy session c902805060df4df9b5599d0e94b2dc06-bad03a4ec8494938839d7d6eef8c684e on tablet bad03a4ec8494938839d7d6eef8c684e with peer c902805060df4df9b5599d0e94b2dc06
I20240502 14:32:30.708001 8172 tablet_bootstrap.cc:492] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:32:30.708480 8172 tablet_bootstrap.cc:492] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4: Bootstrap complete.
I20240502 14:32:30.708873 8172 ts_tablet_manager.cc:1367] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4: Time spent bootstrapping tablet: real 0.075s user 0.032s sys 0.004s
I20240502 14:32:30.709885 8172 raft_consensus.cc:348] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:32:30.710239 8172 raft_consensus.cc:729] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Initialized, Role: FOLLOWER
I20240502 14:32:30.710565 8158 tablet_bootstrap.cc:492] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:32:30.710536 8172 consensus_queue.cc:260] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:32:30.711123 8158 tablet_bootstrap.cc:492] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b: Bootstrap complete.
I20240502 14:32:30.711256 8172 ts_tablet_manager.cc:1398] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:32:30.711477 8158 ts_tablet_manager.cc:1367] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b: Time spent bootstrapping tablet: real 0.052s user 0.024s sys 0.006s
I20240502 14:32:30.712193 19116 tablet_copy_service.cc:342] P 007b2e79dc424a6f80141180f072f2d8: Request end of tablet copy session fec04c50dc5b49878ffa3c3873df32e4-e5fb3b049ba94af998791640bd0e31c5 received from {username='slave'} at 127.5.107.66:33187
I20240502 14:32:30.712370 19116 tablet_copy_service.cc:433] P 007b2e79dc424a6f80141180f072f2d8: ending tablet copy session fec04c50dc5b49878ffa3c3873df32e4-e5fb3b049ba94af998791640bd0e31c5 on tablet e5fb3b049ba94af998791640bd0e31c5 with peer fec04c50dc5b49878ffa3c3873df32e4
I20240502 14:32:30.712237 8158 raft_consensus.cc:348] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:32:30.712603 8158 raft_consensus.cc:729] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Initialized, Role: FOLLOWER
I20240502 14:32:30.712898 8158 consensus_queue.cc:260] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 3.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:32:30.714285 8158 ts_tablet_manager.cc:1398] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:32:30.714991 20647 tablet_copy_service.cc:342] P a834edffbcb2469aae598b069aab0681: Request end of tablet copy session 2db304b538e14857b7b1cc272d0db94b-522014370f0a4cc3bd31aa00d6a05c5e received from {username='slave'} at 127.5.107.65:53335
I20240502 14:32:30.715121 20647 tablet_copy_service.cc:433] P a834edffbcb2469aae598b069aab0681: ending tablet copy session 2db304b538e14857b7b1cc272d0db94b-522014370f0a4cc3bd31aa00d6a05c5e on tablet 522014370f0a4cc3bd31aa00d6a05c5e with peer 2db304b538e14857b7b1cc272d0db94b
W20240502 14:32:30.760282 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:30.769430 19143 maintenance_manager.cc:419] P 007b2e79dc424a6f80141180f072f2d8: Scheduling UndoDeltaBlockGCOp(21af917743444f2db932636a78f33512): 4840 bytes on disk
I20240502 14:32:30.770232 19033 maintenance_manager.cc:643] P 007b2e79dc424a6f80141180f072f2d8: UndoDeltaBlockGCOp(21af917743444f2db932636a78f33512) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240502 14:32:30.771028 18748 maintenance_manager.cc:419] P 2db304b538e14857b7b1cc272d0db94b: Scheduling UndoDeltaBlockGCOp(522014370f0a4cc3bd31aa00d6a05c5e): 4840 bytes on disk
I20240502 14:32:30.771716 18638 maintenance_manager.cc:643] P 2db304b538e14857b7b1cc272d0db94b: UndoDeltaBlockGCOp(522014370f0a4cc3bd31aa00d6a05c5e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":121,"lbm_reads_lt_1ms":4}
I20240502 14:32:30.773219 18748 maintenance_manager.cc:419] P 2db304b538e14857b7b1cc272d0db94b: Scheduling UndoDeltaBlockGCOp(6acde0e5509340fbb52bca18645a77b7): 4586 bytes on disk
I20240502 14:32:30.773921 18638 maintenance_manager.cc:643] P 2db304b538e14857b7b1cc272d0db94b: UndoDeltaBlockGCOp(6acde0e5509340fbb52bca18645a77b7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240502 14:32:30.791152 18880 maintenance_manager.cc:419] P fec04c50dc5b49878ffa3c3873df32e4: Scheduling UndoDeltaBlockGCOp(e5fb3b049ba94af998791640bd0e31c5): 4840 bytes on disk
I20240502 14:32:30.791779 18770 maintenance_manager.cc:643] P fec04c50dc5b49878ffa3c3873df32e4: UndoDeltaBlockGCOp(e5fb3b049ba94af998791640bd0e31c5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240502 14:32:30.803558 8108 raft_consensus.cc:998] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8: Attempting to remove follower a14c220ad2cc4bf9bd7dad65f4803eb0 from the Raft config. Reason: Leader has been unable to successfully communicate with peer a14c220ad2cc4bf9bd7dad65f4803eb0 for more than 300 seconds (300.227s)
I20240502 14:32:30.804283 8108 consensus_queue.cc:237] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:32:30.834420 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:30.842669 18701 raft_consensus.cc:1261] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:32:30.843741 8102 consensus_queue.cc:1035] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:32:30.850891 20626 raft_consensus.cc:1261] T 6a992d12aee7416096d1353533397386 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:32:30.851564 8102 consensus_queue.cc:1035] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 14:32:30.859095 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:30.859181 19012 maintenance_manager.cc:419] P c902805060df4df9b5599d0e94b2dc06: Scheduling UndoDeltaBlockGCOp(bad03a4ec8494938839d7d6eef8c684e): 4840 bytes on disk
I20240502 14:32:30.859805 18901 maintenance_manager.cc:643] P c902805060df4df9b5599d0e94b2dc06: UndoDeltaBlockGCOp(bad03a4ec8494938839d7d6eef8c684e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20240502 14:32:30.862001 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:30.866168 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:30.869475 19095 raft_consensus.cc:1203] T 21af917743444f2db932636a78f33512 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 14:32:30.883941 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:30.910367 8102 raft_consensus.cc:2932] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:32:30.916133 18699 raft_consensus.cc:1203] T 6acde0e5509340fbb52bca18645a77b7 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:32:30.920099 18701 raft_consensus.cc:2932] T 6a992d12aee7416096d1353533397386 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:32:30.921767 18963 raft_consensus.cc:1203] T bad03a4ec8494938839d7d6eef8c684e P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:32:30.926081 18555 catalog_manager.cc:5484] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: config changed from index 8 to 9, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:32:30.955571 8158 ts_tablet_manager.cc:902] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b: Initiating tablet copy from peer c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:32:30.956430 20625 raft_consensus.cc:2932] T 6a992d12aee7416096d1353533397386 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:32:30.964766 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:30.992198 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:30.994000 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:32:30.998651 8158 tablet_copy_client.cc:308] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.67:45409
I20240502 14:32:30.999766 18985 tablet_copy_service.cc:140] P c902805060df4df9b5599d0e94b2dc06: Received BeginTabletCopySession request for tablet b15d11209e45473e863766e5cd6e822f from peer 2db304b538e14857b7b1cc272d0db94b ({username='slave'} at 127.5.107.65:55241)
I20240502 14:32:30.999977 18985 tablet_copy_service.cc:161] P c902805060df4df9b5599d0e94b2dc06: Beginning new tablet copy session on tablet b15d11209e45473e863766e5cd6e822f from peer 2db304b538e14857b7b1cc272d0db94b at {username='slave'} at 127.5.107.65:55241: session id = 2db304b538e14857b7b1cc272d0db94b-b15d11209e45473e863766e5cd6e822f
I20240502 14:32:31.013715 18985 tablet_copy_source_session.cc:205] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:32:31.016244 8158 ts_tablet_manager.cc:1875] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:32:31.023900 18701 raft_consensus.cc:1203] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.12->[3.13-3.13] Dedup: 3.13->[]
I20240502 14:32:31.032582 8158 ts_tablet_manager.cc:1888] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 14:32:31.032872 8158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b15d11209e45473e863766e5cd6e822f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:32:31.038919 8158 tablet_copy_client.cc:783] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b: tablet copy: Starting download of 5 data blocks...
I20240502 14:32:31.046752 8158 tablet_copy_client.cc:647] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:32:31.049058 8158 tablet_copy_client.cc:523] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:32:31.054312 8158 tablet_bootstrap.cc:492] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b: Bootstrap starting.
I20240502 14:32:31.079193 8158 tablet_bootstrap.cc:492] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:32:31.079962 8158 tablet_bootstrap.cc:492] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b: Bootstrap complete.
I20240502 14:32:31.080327 8158 ts_tablet_manager.cc:1367] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b: Time spent bootstrapping tablet: real 0.026s user 0.020s sys 0.004s
I20240502 14:32:31.081353 8158 raft_consensus.cc:348] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:32:31.081589 8158 raft_consensus.cc:729] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Initialized, Role: FOLLOWER
I20240502 14:32:31.081929 8158 consensus_queue.cc:260] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:32:31.085023 8158 ts_tablet_manager.cc:1398] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:32:31.087090 18985 tablet_copy_service.cc:342] P c902805060df4df9b5599d0e94b2dc06: Request end of tablet copy session 2db304b538e14857b7b1cc272d0db94b-b15d11209e45473e863766e5cd6e822f received from {username='slave'} at 127.5.107.65:55241
I20240502 14:32:31.087255 18985 tablet_copy_service.cc:433] P c902805060df4df9b5599d0e94b2dc06: ending tablet copy session 2db304b538e14857b7b1cc272d0db94b-b15d11209e45473e863766e5cd6e822f on tablet b15d11209e45473e863766e5cd6e822f with peer 2db304b538e14857b7b1cc272d0db94b
I20240502 14:32:31.124655 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:31.134380 18833 raft_consensus.cc:1203] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:32:31.154597 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:31.159343 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:31.174237 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:31.213935 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:31.264379 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:31.277534 18748 maintenance_manager.cc:419] P 2db304b538e14857b7b1cc272d0db94b: Scheduling UndoDeltaBlockGCOp(b15d11209e45473e863766e5cd6e822f): 4925 bytes on disk
I20240502 14:32:31.278157 18638 maintenance_manager.cc:643] P 2db304b538e14857b7b1cc272d0db94b: UndoDeltaBlockGCOp(b15d11209e45473e863766e5cd6e822f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
W20240502 14:32:31.319717 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:32:31.351631 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:31.381265 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:31.394042 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:31.397928 18701 raft_consensus.cc:1203] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240502 14:32:31.409632 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:31.414378 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:31.444496 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:31.468868 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:31.472869 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:31.631316 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:31.652372 19093 consensus_queue.cc:237] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:32:31.663002 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:31.684098 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:31.691162 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:31.713497 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:31.737143 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:31.741250 18833 raft_consensus.cc:1261] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:32:31.741827 8102 consensus_queue.cc:1035] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 14:32:31.752673 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:31.755960 18635 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:32:31.769110 8360 raft_consensus.cc:2932] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } }
W20240502 14:32:31.786232 20562 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:32:31.786912 18833 raft_consensus.cc:2932] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } }
W20240502 14:32:31.789126 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:31.799919 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:32:31.801033 18560 catalog_manager.cc:5484] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index 7 to 8, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) evicted. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } }
I20240502 14:32:31.819988 19093 consensus_queue.cc:237] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
W20240502 14:32:31.822299 19031 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5fb3b049ba94af998791640bd0e31c5. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:32:31.822346 18833 raft_consensus.cc:1261] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:32:31.822880 8361 consensus_queue.cc:1035] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:32:31.825095 8359 raft_consensus.cc:2932] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
W20240502 14:32:31.826547 19032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240502 14:32:31.828419 18681 tablet_service.cc:1514] Processing DeleteTablet for tablet e5fb3b049ba94af998791640bd0e31c5 with delete_type TABLET_DATA_TOMBSTONED (TS 2db304b538e14857b7b1cc272d0db94b not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:33750
I20240502 14:32:31.829661 8365 tablet_replica.cc:330] T e5fb3b049ba94af998791640bd0e31c5 P 2db304b538e14857b7b1cc272d0db94b: stopping tablet replica
I20240502 14:32:31.829865 8365 raft_consensus.cc:2229] T e5fb3b049ba94af998791640bd0e31c5 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:32:31.830117 8365 raft_consensus.cc:2258] T e5fb3b049ba94af998791640bd0e31c5 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:32:31.830623 18833 raft_consensus.cc:2932] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:32:31.831774 8365 ts_tablet_manager.cc:1875] T e5fb3b049ba94af998791640bd0e31c5 P 2db304b538e14857b7b1cc272d0db94b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:32:31.834112 18546 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet e5fb3b049ba94af998791640bd0e31c5 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
W20240502 14:32:31.839269 18544 catalog_manager.cc:4573] TS a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5be0d6f4f1874c79b906c4bdc34b0ce9: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:31.840369 18560 catalog_manager.cc:5484] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: config changed from index 8 to 9, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) added. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:32:31.841244 8365 ts_tablet_manager.cc:1888] T e5fb3b049ba94af998791640bd0e31c5 P 2db304b538e14857b7b1cc272d0db94b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 14:32:31.841363 8365 log.cc:1196] T e5fb3b049ba94af998791640bd0e31c5 P 2db304b538e14857b7b1cc272d0db94b: Deleting WAL directory at /tmp/dist-test-taskgPEJ6s/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714659790480533-5549-0/raft_consensus-itest-cluster/ts-0/wal/wals/e5fb3b049ba94af998791640bd0e31c5
I20240502 14:32:31.842068 18544 catalog_manager.cc:4830] TS 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749): tablet e5fb3b049ba94af998791640bd0e31c5 (table rebalance_test_table_6 [id=995bc008b9f94332bfa2c42a77e41bb2]) successfully deleted
W20240502 14:32:31.887428 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:32:31.990770 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:32:31.996132 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:32:32.033638 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 14:32:32.082194 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 14:32:32.087481 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:32:32.150154 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:32.162212 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:32.194924 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:32.195888 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:32.205808 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:32.215777 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:32.263013 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:32.286055 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:32.319202 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:32.329180 8448 ts_tablet_manager.cc:902] T e5fb3b049ba94af998791640bd0e31c5 P ff6acba046404ef0a6f41ad5e09d852f: Initiating tablet copy from peer 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463)
F20240502 14:32:32.329528 8448 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-097eb705c9e24740a5be61bf3354f3d8.queue_time_us, tablet-download-pool-097eb705c9e24740a5be61bf3354f3d8.queue_time_us) (tablet-download-pool-097eb705c9e24740a5be61bf3354f3d8.run_cpu_time_us, tablet-download-pool-097eb705c9e24740a5be61bf3354f3d8.run_cpu_time_us) (tablet-download-pool-097eb705c9e24740a5be61bf3354f3d8.run_wall_time_us, tablet-download-pool-097eb705c9e24740a5be61bf3354f3d8.run_wall_time_us) (tablet-download-pool-0d8667d91fd64809b635362c4a8fa33f.queue_time_us, tablet-download-pool-0d8667d91fd64809b635362c4a8fa33f.queue_time_us) (tablet-download-pool-0d8667d91fd64809b635362c4a8fa33f.run_cpu_time_us, tablet-download-pool-0d8667d91fd64809b635362c4a8fa33f.run_cpu_time_us) (tablet-download-pool-0d8667d91fd64809b635362c4a8fa33f.run_wall_time_us, tablet-download-pool-0d8667d91fd64809b635362c4a8fa33f.run_wall_time_us) (tablet-download-pool-0ffb6e582ef04fb38dd273c13febac58.queue_time_us, tablet-download-pool-0ffb6e582ef04fb38dd273c13febac58.queue_time_us) (tablet-download-pool-0ffb6e582ef04fb38dd273c13febac58.run_cpu_time_us, tablet-download-pool-0ffb6e582ef04fb38dd273c13febac58.run_cpu_time_us) (tablet-download-pool-0ffb6e582ef04fb38dd273c13febac58.run_wall_time_us, tablet-download-pool-0ffb6e582ef04fb38dd273c13febac58.run_wall_time_us) (tablet-download-pool-1ecc6539b85d40e49df7ffc5f93cb801.queue_time_us, tablet-download-pool-1ecc6539b85d40e49df7ffc5f93cb801.queue_time_us) (tablet-download-pool-1ecc6539b85d40e49df7ffc5f93cb801.run_cpu_time_us, tablet-download-pool-1ecc6539b85d40e49df7ffc5f93cb801.run_cpu_time_us) (tablet-download-pool-1ecc6539b85d40e49df7ffc5f93cb801.run_wall_time_us, tablet-download-pool-1ecc6539b85d40e49df7ffc5f93cb801.run_wall_time_us) (tablet-download-pool-2c2afaffde96462eacd0d32bbdec23b1.queue_time_us, tablet-download-pool-2c2afaffde96462eacd0d32bbdec23b1.queue_time_us) (tablet-download-pool-2c2afaffde96462eacd0d32bbdec23b1.run_cpu_time_us, tablet-download-pool-2c2afaffde96462eacd0d32bbdec23b1.run_cpu_time_us) (tablet-download-pool-2c2afaffde96462eacd0d32bbdec23b1.run_wall_time_us, tablet-download-pool-2c2afaffde96462eacd0d32bbdec23b1.run_wall_time_us) (tablet-download-pool-332e4b23751b422ab7bd17b80c93701d.queue_time_us, tablet-download-pool-332e4b23751b422ab7bd17b80c93701d.queue_time_us) (tablet-download-pool-332e4b23751b422ab7bd17b80c93701d.run_cpu_time_us, tablet-download-pool-332e4b23751b422ab7bd17b80c93701d.run_cpu_time_us) (tablet-download-pool-332e4b23751b422ab7bd17b80c93701d.run_wall_time_us, tablet-download-pool-332e4b23751b422ab7bd17b80c93701d.run_wall_time_us) (tablet-download-pool-51ef0be93c914e258635e13f3cca79aa.queue_time_us, tablet-download-pool-51ef0be93c914e258635e13f3cca79aa.queue_time_us) (tablet-download-pool-51ef0be93c914e258635e13f3cca79aa.run_cpu_time_us, tablet-download-pool-51ef0be93c914e258635e13f3cca79aa.run_cpu_time_us) (tablet-download-pool-51ef0be93c914e258635e13f3cca79aa.run_wall_time_us, tablet-download-pool-51ef0be93c914e258635e13f3cca79aa.run_wall_time_us) (tablet-download-pool-522014370f0a4cc3bd31aa00d6a05c5e.queue_time_us, tablet-download-pool-522014370f0a4cc3bd31aa00d6a05c5e.queue_time_us) (tablet-download-pool-522014370f0a4cc3bd31aa00d6a05c5e.run_cpu_time_us, tablet-download-pool-522014370f0a4cc3bd31aa00d6a05c5e.run_cpu_time_us) (tablet-download-pool-522014370f0a4cc3bd31aa00d6a05c5e.run_wall_time_us, tablet-download-pool-522014370f0a4cc3bd31aa00d6a05c5e.run_wall_time_us) (tablet-download-pool-8937c809610542d88fdde3daf002c890.queue_time_us, tablet-download-pool-8937c809610542d88fdde3daf002c890.queue_time_us) (tablet-download-pool-8937c809610542d88fdde3daf002c890.run_cpu_time_us, tablet-download-pool-8937c809610542d88fdde3daf002c890.run_cpu_time_us) (tablet-download-pool-8937c809610542d88fdde3daf002c890.run_wall_time_us, tablet-download-pool-8937c809610542d88fdde3daf002c890.run_wall_time_us) (tablet-download-pool-8b8d81c275144f898dcc69a6d98d11e0.queue_time_us, tablet-download-pool-8b8d81c275144f898dcc69a6d98d11e0.queue_time_us) (tablet-download-pool-8b8d81c275144f898dcc69a6d98d11e0.run_cpu_time_us, tablet-download-pool-8b8d81c275144f898dcc69a6d98d11e0.run_cpu_time_us) (tablet-download-pool-8b8d81c275144f898dcc69a6d98d11e0.run_wall_time_us, tablet-download-pool-8b8d81c275144f898dcc69a6d98d11e0.run_wall_time_us) (tablet-download-pool-9580f3fdf7084cc0923b1b16bfedd450.queue_time_us, tablet-download-pool-9580f3fdf7084cc0923b1b16bfedd450.queue_time_us) (tablet-download-pool-9580f3fdf7084cc0923b1b16bfedd450.run_cpu_time_us, tablet-download-pool-9580f3fdf7084cc0923b1b16bfedd450.run_cpu_time_us) (tablet-download-pool-9580f3fdf7084cc0923b1b16bfedd450.run_wall_time_us, tablet-download-pool-9580f3fdf7084cc0923b1b16bfedd450.run_wall_time_us) (tablet-download-pool-b15d11209e45473e863766e5cd6e822f.queue_time_us, tablet-download-pool-b15d11209e45473e863766e5cd6e822f.queue_time_us) (tablet-download-pool-b15d11209e45473e863766e5cd6e822f.run_cpu_time_us, tablet-download-pool-b15d11209e45473e863766e5cd6e822f.run_cpu_time_us) (tablet-download-pool-b15d11209e45473e863766e5cd6e822f.run_wall_time_us, tablet-download-pool-b15d11209e45473e863766e5cd6e822f.run_wall_time_us) (tablet-download-pool-b315327141954b53a0f87c2d1dc14b9b.queue_time_us, tablet-download-pool-b315327141954b53a0f87c2d1dc14b9b.queue_time_us) (tablet-download-pool-b315327141954b53a0f87c2d1dc14b9b.run_cpu_time_us, tablet-download-pool-b315327141954b53a0f87c2d1dc14b9b.run_cpu_time_us) (tablet-download-pool-b315327141954b53a0f87c2d1dc14b9b.run_wall_time_us, tablet-download-pool-b315327141954b53a0f87c2d1dc14b9b.run_wall_time_us) (tablet-download-pool-db56136f2dbb429098d4d78b15a3a439.queue_time_us, tablet-download-pool-db56136f2dbb429098d4d78b15a3a439.queue_time_us) (tablet-download-pool-db56136f2dbb429098d4d78b15a3a439.run_cpu_time_us, tablet-download-pool-db56136f2dbb429098d4d78b15a3a439.run_cpu_time_us) (tablet-download-pool-db56136f2dbb429098d4d78b15a3a439.run_wall_time_us, tablet-download-pool-db56136f2dbb429098d4d78b15a3a439.run_wall_time_us) (tablet-download-pool-e5fb3b049ba94af998791640bd0e31c5.queue_time_us, tablet-download-pool-e5fb3b049ba94af998791640bd0e31c5.queue_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f65c150adcd google::LogMessage::Fail() at ??:0
@ 0x7f65c150eb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f65c150a7cc google::LogMessage::Flush() at ??:0
W20240502 14:32:32.351871 20964 consensus_peers.cc:462] T 1ecc6539b85d40e49df7ffc5f93cb801 P ff6acba046404ef0a6f41ad5e09d852f -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
@ 0x7f65c150bf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 14:32:32.353274 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
@ 0x7f65c424b742 kudu::TraceMetrics::InternName() at ??:0
I20240502 14:32:32.382642 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
@ 0x7f65c4217f8b kudu::ThreadPool::ThreadPool() at ??:0
I20240502 14:32:32.385352 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:32.389369 21007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
@ 0x7f65c4211cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f65d36648de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f65d367cdaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f65d37a7480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f65d37dfa8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
W20240502 14:32:32.454437 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
@ 0x7f65d37c2981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f65d37c270e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f65c421bb75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f65c421f19c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f65c421f019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f65c41fa01e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f65c757f6db start_thread at ??:0
@ 0x7f65c03bd71f clone at ??:0
I20240502 14:32:32.497334 18963 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b15d11209e45473e863766e5cd6e822f"
dest_uuid: "c902805060df4df9b5599d0e94b2dc06"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40916
I20240502 14:32:32.497525 18963 raft_consensus.cc:595] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [term 2 LEADER]: Received request to transfer leadership
W20240502 14:32:32.500788 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:32:32.503345 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:32:32.604969 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 14:32:32.714883 18681 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 14:32:32.724412 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:32.735817 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:32.759615 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:32.775832 18945 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:32:32.776412 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:32.801666 20561 connection.cc:535] client connection to 127.5.107.72:35871 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:32:32.801918 18768 connection.cc:535] client connection to 127.5.107.72:35871 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:32:32.802062 18899 connection.cc:535] client connection to 127.5.107.72:35871 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:32:32.802140 19031 connection.cc:535] client connection to 127.5.107.72:35871 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:32:32.802603 18636 connection.cc:535] client connection to 127.5.107.72:35871 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:32:32.803467 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:32:32.803550 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:32:32.804133 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:32:32.804349 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:32:32.804456 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:32:32.804782 18899 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:32:32.804997 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:32:32.805102 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:32:32.805233 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:32:32.821672 18769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.69:41119, user_credentials={real_user=slave}} blocked reactor thread for 52985us
W20240502 14:32:32.832556 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:32:32.852182 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:32:32.853415 18963 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b15d11209e45473e863766e5cd6e822f"
dest_uuid: "c902805060df4df9b5599d0e94b2dc06"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40942
I20240502 14:32:32.853655 18963 raft_consensus.cc:595] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [term 2 LEADER]: Received request to transfer leadership
W20240502 14:32:32.855551 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:32:32.921224 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:32:32.966429 8561 raft_consensus.cc:979] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06: : Instructing follower 2db304b538e14857b7b1cc272d0db94b to start an election
I20240502 14:32:32.967388 8561 raft_consensus.cc:1067] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [term 2 LEADER]: Signalling peer 2db304b538e14857b7b1cc272d0db94b to start an election
I20240502 14:32:32.968215 18701 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b15d11209e45473e863766e5cd6e822f"
dest_uuid: "2db304b538e14857b7b1cc272d0db94b"
from {username='slave'} at 127.5.107.67:57951
I20240502 14:32:32.968429 18701 raft_consensus.cc:482] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:32:32.968544 18701 raft_consensus.cc:3037] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:32:32.971884 18701 raft_consensus.cc:504] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:32:32.972604 18701 leader_election.cc:288] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 election: Requested vote from peers c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409), ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871)
I20240502 14:32:32.973492 18963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15d11209e45473e863766e5cd6e822f" candidate_uuid: "2db304b538e14857b7b1cc272d0db94b" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "c902805060df4df9b5599d0e94b2dc06"
I20240502 14:32:32.973678 18963 raft_consensus.cc:3032] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:32:32.973738 18963 raft_consensus.cc:729] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c902805060df4df9b5599d0e94b2dc06, State: Running, Role: LEADER
I20240502 14:32:32.974081 18963 consensus_queue.cc:260] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:32:32.974454 18963 raft_consensus.cc:3037] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [term 2 FOLLOWER]: Advancing to term 3
W20240502 14:32:32.979234 18636 leader_election.cc:334] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:32.980408 18963 raft_consensus.cc:2454] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2db304b538e14857b7b1cc272d0db94b in term 3.
I20240502 14:32:32.980962 18637 leader_election.cc:302] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b, c902805060df4df9b5599d0e94b2dc06; no voters: ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:32:32.981427 8247 raft_consensus.cc:2781] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:32:32.991621 8247 raft_consensus.cc:686] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b [term 3 LEADER]: Becoming Leader. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Running, Role: LEADER
I20240502 14:32:32.991990 8247 consensus_queue.cc:237] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
W20240502 14:32:33.005097 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:32:33.015923 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:32:33.019557 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:33.023294 18557 catalog_manager.cc:5484] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b reported cstate change: term changed from 2 to 3, leader changed from c902805060df4df9b5599d0e94b2dc06 (127.5.107.67) to 2db304b538e14857b7b1cc272d0db94b (127.5.107.65). New cstate: current_term: 3 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:32:33.038247 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:33.071205 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:33.095135 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:33.118183 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:32:33.136987 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:33.171063 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:33.192800 18544 catalog_manager.cc:4573] TS a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6a992d12aee7416096d1353533397386: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:33.197331 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:33.234746 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:33.244088 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:33.249801 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:33.381127 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:32:33.385738 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:33.394441 8247 raft_consensus.cc:998] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b: Attempting to remove follower a14c220ad2cc4bf9bd7dad65f4803eb0 from the Raft config. Reason: Leader has been unable to successfully communicate with peer a14c220ad2cc4bf9bd7dad65f4803eb0 for more than 300 seconds (300.266s)
I20240502 14:32:33.395350 8247 consensus_queue.cc:237] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
W20240502 14:32:33.396363 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240502 14:32:33.410647 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:33.416416 18963 raft_consensus.cc:1261] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 14:32:33.417207 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:32:33.417507 8247 consensus_queue.cc:1035] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:32:33.423699 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:33.447983 18962 raft_consensus.cc:2932] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
I20240502 14:32:33.448621 8623 raft_consensus.cc:2932] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
I20240502 14:32:33.455262 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:33.456447 18553 catalog_manager.cc:5484] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06 reported cstate change: config changed from index 8 to 9, VOTER a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70) evicted. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
I20240502 14:32:33.471416 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:33.490825 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:32:33.508179 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:33.536895 18963 raft_consensus.cc:1261] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [term 3 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 14:32:33.537623 8565 consensus_queue.cc:1035] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 14:32:33.549005 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:33.561111 8623 raft_consensus.cc:998] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b: Attempting to remove follower 7ab6d395b30d4afb9821b61c9454430b from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7ab6d395b30d4afb9821b61c9454430b for more than 300 seconds (300.002s)
I20240502 14:32:33.561864 8623 consensus_queue.cc:237] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:32:33.608093 20626 raft_consensus.cc:1261] T d28db16c9f7e405b8866f0e6c548e246 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:32:33.608238 19093 raft_consensus.cc:1261] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:32:33.608851 8565 consensus_queue.cc:1035] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:32:33.609196 8626 consensus_queue.cc:1035] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:32:33.614665 8626 raft_consensus.cc:2932] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:32:33.619148 20626 raft_consensus.cc:2932] T d28db16c9f7e405b8866f0e6c548e246 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:32:33.620622 19093 raft_consensus.cc:2932] T d28db16c9f7e405b8866f0e6c548e246 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:32:33.621454 18560 catalog_manager.cc:5484] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b reported cstate change: config changed from index 8 to 9, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) evicted. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:32:33.736502 8565 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:32:33.736690 8565 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:32:33.737421 8565 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
W20240502 14:32:33.739575 18636 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:32:33.739780 18637 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:33.739887 18637 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: 7ab6d395b30d4afb9821b61c9454430b, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:32:33.740335 8565 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:32:33.748267 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 14:32:33.749130 8561 raft_consensus.cc:998] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06: Attempting to remove follower 7ab6d395b30d4afb9821b61c9454430b from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7ab6d395b30d4afb9821b61c9454430b for more than 300 seconds (300.075s)
I20240502 14:32:33.749917 8561 consensus_queue.cc:237] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:32:33.758809 20626 raft_consensus.cc:1261] T b13bcd6e693f4c7687a1aabdafbb83c6 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:32:33.760869 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:33.759951 8673 consensus_queue.cc:1035] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240502 14:32:33.784859 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:33.786499 18812 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:32:33.802263 8678 raft_consensus.cc:2932] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:32:33.805593 20626 raft_consensus.cc:2932] T b13bcd6e693f4c7687a1aabdafbb83c6 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:32:33.806097 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:33.815289 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:33.815554 18557 catalog_manager.cc:5484] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 reported cstate change: config changed from index 7 to 8, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) evicted. New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:32:33.822746 18962 consensus_queue.cc:237] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
W20240502 14:32:33.824621 18898 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749): Couldn't send request to peer 2db304b538e14857b7b1cc272d0db94b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:32:33.834867 20627 raft_consensus.cc:1261] T b13bcd6e693f4c7687a1aabdafbb83c6 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:32:33.835585 8679 consensus_queue.cc:1035] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:32:33.846310 8679 raft_consensus.cc:482] T 332e4b23751b422ab7bd17b80c93701d P c902805060df4df9b5599d0e94b2dc06 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:32:33.846446 8679 raft_consensus.cc:504] T 332e4b23751b422ab7bd17b80c93701d P c902805060df4df9b5599d0e94b2dc06 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:32:33.847134 8679 leader_election.cc:288] T 332e4b23751b422ab7bd17b80c93701d P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009), ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871)
I20240502 14:32:33.848583 18833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "332e4b23751b422ab7bd17b80c93701d" candidate_uuid: "c902805060df4df9b5599d0e94b2dc06" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "fec04c50dc5b49878ffa3c3873df32e4" is_pre_election: true
W20240502 14:32:33.849124 18899 leader_election.cc:334] T 332e4b23751b422ab7bd17b80c93701d P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:33.849308 18897 leader_election.cc:302] T 332e4b23751b422ab7bd17b80c93701d P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c902805060df4df9b5599d0e94b2dc06; no voters: fec04c50dc5b49878ffa3c3873df32e4, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:32:33.850979 8679 raft_consensus.cc:2726] T 332e4b23751b422ab7bd17b80c93701d P c902805060df4df9b5599d0e94b2dc06 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:32:33.855075 8679 raft_consensus.cc:2932] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:32:33.858253 20627 raft_consensus.cc:2932] T b13bcd6e693f4c7687a1aabdafbb83c6 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:32:33.858510 8684 raft_consensus.cc:998] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681: Attempting to remove follower 7ab6d395b30d4afb9821b61c9454430b from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7ab6d395b30d4afb9821b61c9454430b for more than 300 seconds (300.321s)
I20240502 14:32:33.859251 8684 consensus_queue.cc:237] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:32:33.861692 8307 raft_consensus.cc:482] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:32:33.861865 8307 raft_consensus.cc:504] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:32:33.861997 18546 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet b13bcd6e693f4c7687a1aabdafbb83c6 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:32:33.862540 8307 leader_election.cc:288] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409), ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871)
I20240502 14:32:33.864197 18962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "332e4b23751b422ab7bd17b80c93701d" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06" is_pre_election: true
I20240502 14:32:33.864374 18962 raft_consensus.cc:2454] T 332e4b23751b422ab7bd17b80c93701d P c902805060df4df9b5599d0e94b2dc06 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 2.
W20240502 14:32:33.864724 18768 leader_election.cc:334] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:33.864881 18769 leader_election.cc:302] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c902805060df4df9b5599d0e94b2dc06, fec04c50dc5b49878ffa3c3873df32e4; no voters: ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:32:33.865234 8307 raft_consensus.cc:2781] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 14:32:33.865332 8307 raft_consensus.cc:482] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:32:33.865434 8307 raft_consensus.cc:3037] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:32:33.868131 18557 catalog_manager.cc:5484] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 reported cstate change: config changed from index 8 to 9, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) added. New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:32:33.868453 8307 raft_consensus.cc:504] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:32:33.869383 18962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "332e4b23751b422ab7bd17b80c93701d" candidate_uuid: "fec04c50dc5b49878ffa3c3873df32e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "c902805060df4df9b5599d0e94b2dc06"
I20240502 14:32:33.869596 18962 raft_consensus.cc:3037] T 332e4b23751b422ab7bd17b80c93701d P c902805060df4df9b5599d0e94b2dc06 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:32:33.872625 18962 raft_consensus.cc:2454] T 332e4b23751b422ab7bd17b80c93701d P c902805060df4df9b5599d0e94b2dc06 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fec04c50dc5b49878ffa3c3873df32e4 in term 3.
I20240502 14:32:33.873119 18769 leader_election.cc:302] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c902805060df4df9b5599d0e94b2dc06, fec04c50dc5b49878ffa3c3873df32e4; no voters:
W20240502 14:32:33.877791 18768 leader_election.cc:334] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:33.878185 8307 leader_election.cc:288] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 [CANDIDATE]: Term 3 election: Requested vote from peers c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409), ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871)
I20240502 14:32:33.878361 8307 raft_consensus.cc:2781] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:32:33.878474 8307 raft_consensus.cc:686] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 [term 3 LEADER]: Becoming Leader. State: Replica: fec04c50dc5b49878ffa3c3873df32e4, State: Running, Role: LEADER
I20240502 14:32:33.878737 8307 consensus_queue.cc:237] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:32:33.898607 18962 raft_consensus.cc:1261] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06 [term 2 FOLLOWER]: Refusing update from remote peer a834edffbcb2469aae598b069aab0681: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:32:33.899148 18557 catalog_manager.cc:5484] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: term changed from 2 to 3, leader changed from ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) to fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66). New cstate: current_term: 3 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
W20240502 14:32:33.904544 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 14:32:33.907536 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:32:33.909145 8683 consensus_queue.cc:1035] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:32:33.921391 8678 raft_consensus.cc:998] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06: Attempting to remove follower 7ab6d395b30d4afb9821b61c9454430b from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7ab6d395b30d4afb9821b61c9454430b for more than 300 seconds (300.049s)
I20240502 14:32:33.922222 8678 consensus_queue.cc:237] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:32:33.926272 20627 raft_consensus.cc:1261] T ecb8a9b04ba34f779e2227e77e6f764c P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:32:33.926985 8561 consensus_queue.cc:1035] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:32:33.939126 8683 raft_consensus.cc:2932] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:32:33.939724 18962 raft_consensus.cc:2932] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:32:33.947033 18557 catalog_manager.cc:5484] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 reported cstate change: config changed from index 9 to 10, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) evicted. New cstate: current_term: 2 leader_uuid: "a834edffbcb2469aae598b069aab0681" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:32:33.948185 8561 raft_consensus.cc:2932] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:32:33.948688 20627 raft_consensus.cc:2932] T ecb8a9b04ba34f779e2227e77e6f764c P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:32:33.959942 18557 catalog_manager.cc:5484] T ecb8a9b04ba34f779e2227e77e6f764c P a834edffbcb2469aae598b069aab0681 reported cstate change: config changed from index 7 to 8, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) evicted. New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
W20240502 14:32:33.961685 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:33.967728 18962 consensus_queue.cc:237] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
W20240502 14:32:33.969799 18898 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749): Couldn't send request to peer 2db304b538e14857b7b1cc272d0db94b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:32:33.970484 8361 raft_consensus.cc:998] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8: Attempting to remove follower 7ab6d395b30d4afb9821b61c9454430b from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7ab6d395b30d4afb9821b61c9454430b for more than 300 seconds (300.351s)
I20240502 14:32:33.971256 8361 consensus_queue.cc:237] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:32:33.972756 20627 raft_consensus.cc:1261] T ecb8a9b04ba34f779e2227e77e6f764c P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:32:33.974368 8678 consensus_queue.cc:1035] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:32:33.974779 20627 raft_consensus.cc:1261] T 8937c809610542d88fdde3daf002c890 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:32:33.975561 8701 consensus_queue.cc:1035] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 14:32:33.975812 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:32:33.977164 8678 raft_consensus.cc:2932] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:32:33.979300 8701 raft_consensus.cc:2932] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:32:33.984068 20627 raft_consensus.cc:2932] T ecb8a9b04ba34f779e2227e77e6f764c P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:32:33.984028 20625 raft_consensus.cc:2932] T 8937c809610542d88fdde3daf002c890 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:32:33.987125 8689 raft_consensus.cc:998] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4: Attempting to remove follower 7ab6d395b30d4afb9821b61c9454430b from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7ab6d395b30d4afb9821b61c9454430b for more than 300 seconds (300.201s)
I20240502 14:32:33.986922 18560 catalog_manager.cc:5484] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: config changed from index 8 to 9, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) evicted. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:32:33.987854 8689 consensus_queue.cc:237] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:32:33.991838 18557 catalog_manager.cc:5484] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 reported cstate change: config changed from index 8 to 9, VOTER 2db304b538e14857b7b1cc272d0db94b (127.5.107.65) added. New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
W20240502 14:32:33.992748 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:33.993366 18546 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet ecb8a9b04ba34f779e2227e77e6f764c with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:32:34.008560 20627 raft_consensus.cc:1261] T 9580f3fdf7084cc0923b1b16bfedd450 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 14:32:34.008569 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:32:34.009168 8307 consensus_queue.cc:1035] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:32:34.013275 8689 raft_consensus.cc:2932] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:32:34.013810 20627 raft_consensus.cc:2932] T 9580f3fdf7084cc0923b1b16bfedd450 P a834edffbcb2469aae598b069aab0681 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:32:34.018769 18553 catalog_manager.cc:5484] T 9580f3fdf7084cc0923b1b16bfedd450 P a834edffbcb2469aae598b069aab0681 reported cstate change: config changed from index 8 to 9, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) evicted. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } }
I20240502 14:32:34.114799 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:32:34.122922 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 14:32:34.130731 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:34.137876 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:34.139025 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:34.140727 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:34.218530 18546 catalog_manager.cc:4573] TS 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8937c809610542d88fdde3daf002c890: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:34.246387 8752 ts_tablet_manager.cc:902] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b: Initiating tablet copy from peer c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
I20240502 14:32:34.263135 8752 tablet_copy_client.cc:308] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.67:45409
I20240502 14:32:34.263981 18985 tablet_copy_service.cc:140] P c902805060df4df9b5599d0e94b2dc06: Received BeginTabletCopySession request for tablet b13bcd6e693f4c7687a1aabdafbb83c6 from peer 2db304b538e14857b7b1cc272d0db94b ({username='slave'} at 127.5.107.65:55241)
I20240502 14:32:34.264151 18985 tablet_copy_service.cc:161] P c902805060df4df9b5599d0e94b2dc06: Beginning new tablet copy session on tablet b13bcd6e693f4c7687a1aabdafbb83c6 from peer 2db304b538e14857b7b1cc272d0db94b at {username='slave'} at 127.5.107.65:55241: session id = 2db304b538e14857b7b1cc272d0db94b-b13bcd6e693f4c7687a1aabdafbb83c6
I20240502 14:32:34.267567 18985 tablet_copy_source_session.cc:205] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:32:34.268764 8752 ts_tablet_manager.cc:1875] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:32:34.284987 8752 ts_tablet_manager.cc:1888] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 14:32:34.285259 8752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b13bcd6e693f4c7687a1aabdafbb83c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:32:34.288179 8752 tablet_copy_client.cc:783] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b: tablet copy: Starting download of 5 data blocks...
I20240502 14:32:34.294057 8752 tablet_copy_client.cc:647] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:32:34.295850 8752 tablet_copy_client.cc:523] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:32:34.299345 8752 tablet_bootstrap.cc:492] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b: Bootstrap starting.
I20240502 14:32:34.319593 8752 tablet_bootstrap.cc:492] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:32:34.320061 8752 tablet_bootstrap.cc:492] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b: Bootstrap complete.
I20240502 14:32:34.320386 8752 ts_tablet_manager.cc:1367] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b: Time spent bootstrapping tablet: real 0.021s user 0.022s sys 0.000s
I20240502 14:32:34.321159 8752 raft_consensus.cc:348] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:32:34.321410 8752 raft_consensus.cc:729] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Initialized, Role: FOLLOWER
I20240502 14:32:34.321681 8752 consensus_queue.cc:260] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:32:34.322532 8752 ts_tablet_manager.cc:1398] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:32:34.325654 18985 tablet_copy_service.cc:342] P c902805060df4df9b5599d0e94b2dc06: Request end of tablet copy session 2db304b538e14857b7b1cc272d0db94b-b13bcd6e693f4c7687a1aabdafbb83c6 received from {username='slave'} at 127.5.107.65:55241
I20240502 14:32:34.325780 18985 tablet_copy_service.cc:433] P c902805060df4df9b5599d0e94b2dc06: ending tablet copy session 2db304b538e14857b7b1cc272d0db94b-b13bcd6e693f4c7687a1aabdafbb83c6 on tablet b13bcd6e693f4c7687a1aabdafbb83c6 with peer 2db304b538e14857b7b1cc272d0db94b
W20240502 14:32:34.337481 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240502 14:32:34.342273 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:32:34.351209 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:34.368247 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:34.376183 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:34.376924 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:32:34.379676 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:34.380302 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:34.485685 18962 raft_consensus.cc:1261] T 332e4b23751b422ab7bd17b80c93701d P c902805060df4df9b5599d0e94b2dc06 [term 3 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240502 14:32:34.486202 8689 consensus_queue.cc:1035] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:32:34.542256 18748 maintenance_manager.cc:419] P 2db304b538e14857b7b1cc272d0db94b: Scheduling UndoDeltaBlockGCOp(b13bcd6e693f4c7687a1aabdafbb83c6): 4925 bytes on disk
I20240502 14:32:34.542896 18638 maintenance_manager.cc:643] P 2db304b538e14857b7b1cc272d0db94b: UndoDeltaBlockGCOp(b13bcd6e693f4c7687a1aabdafbb83c6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
W20240502 14:32:34.574043 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:34.579807 8752 ts_tablet_manager.cc:902] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b: Initiating tablet copy from peer c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
W20240502 14:32:34.612296 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:34.621389 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240502 14:32:34.621779 8752 tablet_copy_client.cc:308] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.67:45409
I20240502 14:32:34.623875 18985 tablet_copy_service.cc:140] P c902805060df4df9b5599d0e94b2dc06: Received BeginTabletCopySession request for tablet ecb8a9b04ba34f779e2227e77e6f764c from peer 2db304b538e14857b7b1cc272d0db94b ({username='slave'} at 127.5.107.65:55241)
I20240502 14:32:34.624034 18985 tablet_copy_service.cc:161] P c902805060df4df9b5599d0e94b2dc06: Beginning new tablet copy session on tablet ecb8a9b04ba34f779e2227e77e6f764c from peer 2db304b538e14857b7b1cc272d0db94b at {username='slave'} at 127.5.107.65:55241: session id = 2db304b538e14857b7b1cc272d0db94b-ecb8a9b04ba34f779e2227e77e6f764c
I20240502 14:32:34.626242 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:34.629478 18985 tablet_copy_source_session.cc:205] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:32:34.630901 8752 ts_tablet_manager.cc:1875] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b: Deleting tablet data with delete state TABLET_DATA_COPYING
W20240502 14:32:34.644348 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:32:34.645814 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:34.646587 18546 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 144 similar messages]
I20240502 14:32:34.654373 8752 ts_tablet_manager.cc:1888] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240502 14:32:34.654661 8752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecb8a9b04ba34f779e2227e77e6f764c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 14:32:34.655894 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:32:34.660193 8752 tablet_copy_client.cc:783] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b: tablet copy: Starting download of 5 data blocks...
I20240502 14:32:34.667065 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:34.672878 8752 tablet_copy_client.cc:647] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:32:34.674487 8752 tablet_copy_client.cc:523] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:32:34.678421 8752 tablet_bootstrap.cc:492] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b: Bootstrap starting.
I20240502 14:32:34.688607 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:34.690047 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:34.749790 8752 tablet_bootstrap.cc:492] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:32:34.760092 8752 tablet_bootstrap.cc:492] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b: Bootstrap complete.
I20240502 14:32:34.760776 8752 ts_tablet_manager.cc:1367] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b: Time spent bootstrapping tablet: real 0.082s user 0.022s sys 0.021s
I20240502 14:32:34.761945 8752 raft_consensus.cc:348] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:32:34.762287 8752 raft_consensus.cc:729] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db304b538e14857b7b1cc272d0db94b, State: Initialized, Role: FOLLOWER
I20240502 14:32:34.762598 8752 consensus_queue.cc:260] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:32:34.763283 8752 ts_tablet_manager.cc:1398] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:32:34.765834 18985 tablet_copy_service.cc:342] P c902805060df4df9b5599d0e94b2dc06: Request end of tablet copy session 2db304b538e14857b7b1cc272d0db94b-ecb8a9b04ba34f779e2227e77e6f764c received from {username='slave'} at 127.5.107.65:55241
I20240502 14:32:34.765961 18985 tablet_copy_service.cc:433] P c902805060df4df9b5599d0e94b2dc06: ending tablet copy session 2db304b538e14857b7b1cc272d0db94b-ecb8a9b04ba34f779e2227e77e6f764c on tablet ecb8a9b04ba34f779e2227e77e6f764c with peer 2db304b538e14857b7b1cc272d0db94b
I20240502 14:32:34.795775 18748 maintenance_manager.cc:419] P 2db304b538e14857b7b1cc272d0db94b: Scheduling UndoDeltaBlockGCOp(ecb8a9b04ba34f779e2227e77e6f764c): 4925 bytes on disk
I20240502 14:32:34.796501 18638 maintenance_manager.cc:643] P 2db304b538e14857b7b1cc272d0db94b: UndoDeltaBlockGCOp(ecb8a9b04ba34f779e2227e77e6f764c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240502 14:32:34.826469 18701 raft_consensus.cc:1203] T b13bcd6e693f4c7687a1aabdafbb83c6 P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
W20240502 14:32:34.879495 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:32:34.900929 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:32:34.902320 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:34.910676 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:34.922749 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:34.925961 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:34.930897 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:35.043754 18699 raft_consensus.cc:1203] T ecb8a9b04ba34f779e2227e77e6f764c P 2db304b538e14857b7b1cc272d0db94b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
W20240502 14:32:35.052748 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:32:35.114364 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:35.117380 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:35.127628 18549 maintenance_manager.cc:419] P 0283476f38234a96bfe72594576461be: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7341 bytes on disk
I20240502 14:32:35.133360 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:35.136933 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:35.146498 18547 maintenance_manager.cc:643] P 0283476f38234a96bfe72594576461be: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.019s user 0.007s sys 0.008s Metrics: {"dirs.queue_time_us":11766,"dirs.run_cpu_time_us":118,"dirs.run_wall_time_us":118,"thread_start_us":12220,"threads_started":2,"wal-append.queue_time_us":301}
I20240502 14:32:35.147598 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:35.154035 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:32:35.158702 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:32:35.164661 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:35.208252 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 14:32:35.213984 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:32:35.239367 18546 catalog_manager.cc:4573] TS 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8937c809610542d88fdde3daf002c890: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:35.270481 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:32:35.333335 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:35.335664 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:32:35.339720 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240502 14:32:35.351580 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:35.351619 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:35.357688 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:35.536159 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:32:35.563927 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:35.568969 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:35.571590 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:35.581540 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:35.589026 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:32:35.616986 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:32:35.688081 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:32:35.689162 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:32:35.697741 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:35.808754 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:35.829152 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:35.837651 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:35.843501 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:35.844386 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:35.854525 8918 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:32:35.854693 8918 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:32:35.855350 8918 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
W20240502 14:32:35.857650 18637 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:32:35.881673 18636 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:35.881896 18636 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: 7ab6d395b30d4afb9821b61c9454430b, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:32:35.882269 8918 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:32:35.978514 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:32:36.038501 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:36.051458 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:36.054131 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:36.055780 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:36.065548 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:36.104220 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:36.192451 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:32:36.283874 18546 catalog_manager.cc:4573] TS 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9580f3fdf7084cc0923b1b16bfedd450: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:36.289898 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:36.292104 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:36.296864 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:32:36.298240 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:36.303308 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:36.334645 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:32:36.348415 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:36.356607 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240502 14:32:36.364857 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:32:36.508878 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:36.518205 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:36.529455 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:36.529633 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:36.547964 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:36.571184 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:32:36.666676 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:36.667990 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:36.747570 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:36.748085 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:36.753968 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:36.779072 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:36.785734 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240502 14:32:36.794100 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:36.811254 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:32:36.837752 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240502 14:32:36.865754 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:32:36.915335 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:32:37.009007 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:37.009474 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:37.025022 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:37.029796 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:37.035234 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:37.054908 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:37.102890 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240502 14:32:37.148458 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:37.207008 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:37.216075 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:37.222463 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:37.224505 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:37.239009 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:37.385465 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:32:37.387336 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:32:37.402580 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:32:37.417258 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:37.422688 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:37.428175 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:32:37.440570 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:37.442046 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:37.459812 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:32:37.463886 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:37.601747 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:37.643841 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:37.648260 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:37.651252 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:37.665472 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:37.666582 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:37.672653 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:37.703366 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240502 14:32:37.756959 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:32:37.776854 18544 catalog_manager.cc:4573] TS a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 0ffb6e582ef04fb38dd273c13febac58: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:37.803390 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:32:37.824472 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:32:37.854100 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:37.860607 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:37.873378 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:37.878532 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:37.879885 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:38.030627 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:32:38.102434 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:32:38.117468 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:38.122519 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:38.129868 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:38.138269 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:38.147248 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:38.152674 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:38.280304 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:38.316334 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:38.325721 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:38.344585 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:38.356822 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:38.364660 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:38.432667 18768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 105 similar messages]
I20240502 14:32:38.529620 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:38.541643 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:38.555135 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:38.555299 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:38.555675 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:38.592622 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:32:38.658509 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:32:38.759405 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:38.759663 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240502 14:32:38.774025 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:38.780810 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:38.782543 18544 catalog_manager.cc:4573] TS a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b15d11209e45473e863766e5cd6e822f: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:38.789255 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:38.790657 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:38.801481 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:32:38.834410 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:32:38.847524 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240502 14:32:38.858243 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:38.949640 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:32:38.957950 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:32:39.019781 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:39.022517 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:39.034658 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:39.046108 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:39.061983 8918 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:32:39.062152 8918 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:32:39.064316 8918 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
W20240502 14:32:39.066038 18636 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:39.069511 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:39.087508 18637 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:39.087646 18637 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: 7ab6d395b30d4afb9821b61c9454430b, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:32:39.088083 8918 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:32:39.174459 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240502 14:32:39.207022 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:39.221550 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:32:39.245492 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:39.248174 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:32:39.257324 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:39.262018 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:39.270781 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:39.284900 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:39.294668 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240502 14:32:39.450665 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:39.452808 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:39.474004 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:39.480302 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:39.488946 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:39.556229 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:32:39.607048 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:32:39.696261 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:39.709931 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:39.721863 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:39.722421 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:39.726198 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240502 14:32:39.727047 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:39.727612 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:39.800590 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:39.913022 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:39.920258 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:39.921237 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:39.926916 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:39.929661 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:39.932196 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:32:39.997450 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:32:40.009166 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:32:40.070530 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:40.109320 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:32:40.131361 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:40.136219 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:40.136619 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:40.149555 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:40.154384 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:40.187206 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240502 14:32:40.194011 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:32:40.213398 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 14:32:40.334141 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:32:40.341217 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:40.350492 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:40.361009 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:40.363343 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:40.382704 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:40.421726 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:32:40.465301 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:32:40.587682 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:40.599225 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:40.601279 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:40.611227 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:40.630251 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:40.667686 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:32:40.669986 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:32:40.673652 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:40.821915 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:40.824446 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:40.827020 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:40.829603 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:40.840552 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:41.022809 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:32:41.022905 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:32:41.026979 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:41.041290 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:41.053628 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 14:32:41.055728 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:41.069628 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:41.073467 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:41.165768 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:32:41.206869 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:32:41.229817 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:41.270740 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:41.280181 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:41.282927 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:41.300369 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:41.304302 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:41.350509 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:32:41.401947 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240502 14:32:41.477597 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:41.477993 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:41.486239 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:41.489555 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:41.490269 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:41.513732 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:32:41.618572 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240502 14:32:41.664188 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:32:41.683020 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:32:41.687997 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:41.697597 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:41.715669 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:41.723217 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:41.750238 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:41.778167 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:41.778882 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 14:32:41.881095 18544 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240502 14:32:41.882705 18544 catalog_manager.cc:4573] TS a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 0ffb6e582ef04fb38dd273c13febac58: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:41.928257 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:41.933744 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:41.933977 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:41.935098 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:41.944636 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:42.055358 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 14:32:42.056217 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:42.151383 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:32:42.156880 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:32:42.157759 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:42.158608 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:42.158617 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:42.185417 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:42.202222 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:32:42.251221 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 14:32:42.254648 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:42.260349 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:32:42.375180 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:42.382560 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:42.384850 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:42.387005 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:42.388661 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:42.546168 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:32:42.560837 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 14:32:42.572254 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:32:42.576179 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:32:42.625470 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:42.641175 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:42.645169 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:42.660179 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:42.661322 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:42.765738 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:32:42.775045 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:32:42.814401 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:32:42.872931 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:42.875838 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:42.879282 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:42.886039 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:42.900442 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:42.901371 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:42.930469 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:32:43.048991 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:32:43.076041 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:32:43.097012 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:43.113153 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:43.125375 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:43.136322 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:43.137120 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:43.255435 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:43.257964 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:32:43.312439 9826 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:32:43.312638 9826 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:32:43.314443 9826 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
W20240502 14:32:43.316972 18636 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:43.318657 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:43.322052 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:32:43.336419 18637 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:43.336649 18637 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: 7ab6d395b30d4afb9821b61c9454430b, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:32:43.337162 9826 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:32:43.345427 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:43.366963 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:43.368317 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:43.440778 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240502 14:32:43.454807 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:43.542847 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:43.543848 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:43.557354 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:43.584175 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:43.587641 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:43.676846 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:32:43.688632 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:32:43.785218 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:43.802260 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:43.803439 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:43.818065 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:43.818459 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:43.832729 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:32:43.832923 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:43.836973 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:32:43.889393 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 14:32:43.899731 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:32:43.900998 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 14:32:44.014665 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:44.025081 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:44.028240 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:44.029549 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:44.035672 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:44.066387 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:32:44.208637 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
I20240502 14:32:44.225306 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:44.229936 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:44.235561 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:32:44.235954 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:44.259373 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:44.260562 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:44.276249 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:44.442822 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
I20240502 14:32:44.459707 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:44.486876 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:44.495944 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:44.502130 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:44.507359 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:44.549551 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:32:44.671872 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240502 14:32:44.677080 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:32:44.710407 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:44.725148 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:44.728356 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:44.728652 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:44.733336 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:32:44.747138 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:32:44.752367 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:44.824574 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:32:44.936427 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:32:44.948326 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:44.953107 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:44.967166 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:44.984301 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:44.985774 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:45.043771 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240502 14:32:45.054092 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:32:45.054433 9826 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:32:45.054584 9826 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:32:45.055202 9826 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
W20240502 14:32:45.056877 18636 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:32:45.069978 18637 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:45.070112 18637 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: 7ab6d395b30d4afb9821b61c9454430b, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:32:45.070333 9826 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:32:45.129472 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:45.172586 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:45.174131 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:45.195154 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:45.195732 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:45.199723 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:45.265077 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:32:45.276840 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 14:32:45.293968 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:45.355438 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:32:45.380404 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:32:45.382071 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:45.387014 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:45.388267 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:45.401880 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:45.402448 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:45.446228 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:32:45.580215 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:45.592914 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:45.605918 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:45.623019 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:45.624378 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:45.742573 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:32:45.768898 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:32:45.788165 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:45.848021 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:45.854555 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:45.855293 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:45.881784 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:45.885406 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:45.993827 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:32:46.020361 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:32:46.047492 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:32:46.062462 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:46.076027 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:46.079221 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:46.083873 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:46.093477 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:46.181469 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:32:46.217489 18544 catalog_manager.cc:4573] TS a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 96662a3d9e1b4f03b05464ddd06b93d6: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:46.234126 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:32:46.270622 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 14:32:46.300884 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:46.312304 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:46.313463 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:46.314982 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:46.329713 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:46.329792 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:46.365845 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:32:46.420312 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:32:46.473963 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:32:46.518983 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:46.520390 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:46.536576 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:46.545343 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:46.551712 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:46.603209 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 14:32:46.618239 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:32:46.651258 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
I20240502 14:32:46.760540 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:46.766677 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:46.769627 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:46.774538 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:46.788988 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:46.854947 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:46.910231 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240502 14:32:46.983038 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:32:46.986567 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:46.990636 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:32:47.013830 18544 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 16 similar messages]
I20240502 14:32:47.014271 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:47.025648 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:47.037660 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:47.046841 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:47.058212 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 14:32:47.089861 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 14:32:47.223053 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:47.231846 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:47.240520 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:47.252818 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:47.257807 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:47.281492 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:47.321770 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:32:47.433082 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:32:47.466861 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:47.478821 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:47.480621 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:47.485565 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:47.485710 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:47.509004 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:32:47.589955 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:32:47.598024 18544 catalog_manager.cc:4573] TS a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6a992d12aee7416096d1353533397386: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:47.603935 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:32:47.621150 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 14:32:47.628065 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:32:47.658123 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:47.685994 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:32:47.701674 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:47.713536 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:47.717698 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:47.729981 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:47.732067 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:47.734798 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:32:47.906271 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:32:47.911477 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:47.911873 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:47.924598 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:47.942116 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:47.942163 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:47.988059 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:32:48.048601 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:48.120640 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:48.129559 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:48.133831 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:48.144448 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:48.146873 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:48.217731 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:32:48.232354 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:32:48.343459 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:48.352053 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:48.359184 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:48.360565 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:48.372217 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:48.469772 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240502 14:32:48.543062 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:48.543048 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:48.572712 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:48.583544 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:48.595371 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:48.595999 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:48.607064 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:48.615228 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:32:48.627338 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:32:48.671973 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:32:48.680533 18636 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.72:35871, user_credentials={real_user=slave}} blocked reactor thread for 53232.7us
I20240502 14:32:48.831292 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:48.848052 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:48.849830 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:48.866436 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:48.869228 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:48.905169 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:32:48.956229 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:32:48.960855 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 14:32:48.962842 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:32:48.966905 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:32:49.070791 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:49.081688 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:49.097620 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:49.099092 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:49.117466 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:49.130350 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:32:49.164577 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:49.291328 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:49.298588 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:49.299273 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:49.322638 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:49.326148 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:49.358676 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 14:32:49.369019 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:32:49.372671 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 14:32:49.380431 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 14:32:49.502482 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:49.513573 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:49.517998 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:49.531759 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:49.532173 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:49.541363 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:49.580976 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:49.737398 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:49.739121 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:49.741693 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:49.750586 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:49.763060 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:49.791956 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 14:32:49.834302 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:32:49.912041 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:32:49.950816 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:49.962016 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:49.963407 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:49.985589 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:50.007508 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:50.019543 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:32:50.022642 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:32:50.057516 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:32:50.114290 18544 catalog_manager.cc:4573] TS a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 0ffb6e582ef04fb38dd273c13febac58: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:50.120987 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:32:50.125674 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:50.153543 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:50.169484 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:50.170816 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:50.186750 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:50.188972 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:50.274827 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:32:50.305261 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 14:32:50.403887 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:50.407789 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:50.415365 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:50.418212 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:50.453632 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:32:50.467551 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:32:50.470687 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:32:50.593869 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:32:50.607498 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:50.634739 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:50.659219 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:32:50.667567 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:50.692943 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:50.694582 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:50.707166 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:50.746629 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:32:50.883926 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:50.895088 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:50.895973 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:50.909035 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:50.910310 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:50.973721 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:32:51.020341 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:51.100554 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:51.106770 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:51.115775 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:51.122633 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:51.127049 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:32:51.129573 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:51.209439 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:32:51.216463 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:32:51.262535 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:32:51.296367 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:32:51.320860 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:51.330307 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:51.337563 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:51.337968 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:51.349941 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:51.407033 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:32:51.413503 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:51.531639 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:51.532552 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:51.550978 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:51.555379 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:51.557444 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:51.577477 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 14:32:51.611522 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:51.642102 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 14:32:51.652523 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 14:32:51.663578 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:32:51.768625 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:51.772684 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:51.774636 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:51.780450 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:51.788439 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:51.822821 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 14:32:51.891211 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
I20240502 14:32:51.974175 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:51.984638 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:52.003729 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:52.029945 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:52.125471 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:52.127022 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:52.140250 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 14:32:52.161778 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:32:52.227397 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:52.288674 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:52.289363 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:52.301663 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:52.310858 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:32:52.317759 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:52.322134 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:52.346493 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 14:32:52.351615 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:32:52.516961 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:52.528034 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:52.528874 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:52.542701 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:52.543118 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:52.546952 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:52.606947 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:32:52.651830 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:52.677703 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:32:52.743973 11808 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:32:52.744195 11808 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:32:52.745044 11808 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
W20240502 14:32:52.747825 18636 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:32:52.748947 18637 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:32:52.749048 18637 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: 7ab6d395b30d4afb9821b61c9454430b, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:32:52.749298 11808 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:32:52.761427 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:52.771247 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:52.773417 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:52.790645 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:52.817694 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:52.895407 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:32:53.021991 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:53.023970 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:53.030259 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:53.040305 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:53.050889 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:53.063376 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:32:53.080122 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:32:53.116766 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:32:53.215286 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:53.229457 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:53.232820 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:53.236876 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:53.241631 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:53.254014 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:32:53.259488 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:53.299290 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:32:53.431210 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:53.434710 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:53.448385 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:53.458945 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:53.459193 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:53.503916 18768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240502 14:32:53.526854 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:32:53.622036 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:32:53.657830 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:32:53.665836 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:53.681125 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:32:53.704967 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:53.705147 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:53.705842 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:53.707947 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:32:53.728097 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:53.799401 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:32:53.854700 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:32:53.908226 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:53.909627 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:53.917503 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:53.920854 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:53.939002 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:54.079974 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 14:32:54.092399 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:54.112229 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:54.112794 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:32:54.114579 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:54.115795 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:54.116529 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 14:32:54.118158 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 14:32:54.118853 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:54.119159 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:32:54.143491 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 14:32:54.198295 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:54.320981 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:32:54.344204 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:54.354506 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:54.357790 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:54.363523 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:54.393956 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:54.568223 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:54.572484 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:54.575098 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:54.577292 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:54.617472 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:54.624850 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:54.654376 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:32:54.700619 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 14:32:54.712469 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:54.788911 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:32:54.817051 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:54.819403 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:54.845523 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:54.845777 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:54.873270 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:54.876672 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:32:54.967101 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:32:55.011173 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 14:32:55.044225 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:32:55.049599 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:32:55.068202 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:55.074800 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:55.100659 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:55.112063 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:55.121147 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:55.170423 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:55.326211 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:55.326596 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:32:55.326890 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 64 similar messages]
I20240502 14:32:55.327961 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:55.340737 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:55.358327 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:55.514902 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:32:55.532145 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:55.555168 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:55.568071 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:55.569940 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:55.571894 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:55.574682 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:55.587307 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:32:55.659533 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:32:55.661290 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:32:55.705142 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:55.775035 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:55.777916 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:55.784694 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:55.795848 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:55.795893 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:55.938901 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:32:55.951370 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:32:55.960060 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:32:55.991611 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:32:55.997014 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:56.013744 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:56.032685 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:56.052861 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:56.058907 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:56.091100 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:32:56.140820 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:32:56.217021 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:56.240177 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:56.241168 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:56.255358 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:56.256242 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:56.274067 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:56.353931 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:32:56.384358 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:56.440183 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:56.453779 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:56.458462 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:56.464516 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:56.474267 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:32:56.632489 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:32:56.653311 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 14:32:56.655161 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 14:32:56.662581 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 14:32:56.678416 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:56.680176 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:56.711838 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:56.713827 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:56.723340 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:56.747439 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:56.806497 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 14:32:56.825575 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:56.884608 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:56.889510 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:56.904367 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:56.908243 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:56.943372 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:56.968128 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:32:57.127761 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:57.134392 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:57.142647 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:57.146550 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:57.157759 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:57.181840 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:57.233517 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:32:57.313668 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:32:57.328725 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:32:57.344082 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:32:57.346796 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:57.346796 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:57.355412 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:57.362345 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:57.362349 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:57.525223 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 14:32:57.536173 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:32:57.547439 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:57.561422 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:57.568022 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:57.574513 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:57.589423 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:57.608460 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:32:57.659780 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:57.792456 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:32:57.821200 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:57.822564 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:57.833156 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:57.845230 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:57.852365 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:57.864262 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:32:57.913816 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 14:32:57.967991 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:58.038908 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:58.053505 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:58.063730 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:58.068799 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:58.077576 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:58.116099 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:32:58.199544 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:32:58.266969 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:32:58.267897 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:58.268397 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:58.272866 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:58.273833 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:58.287098 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:58.287917 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:58.432201 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:32:58.463879 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:32:58.474854 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:32:58.507601 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:32:58.516899 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:58.517140 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:58.518983 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:58.521412 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:58.522749 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:32:58.537896 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:32:58.564491 18768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240502 14:32:58.565650 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:32:58.635167 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:32:58.737959 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:58.738400 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:58.756234 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:32:58.767136 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:58.767205 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:58.790174 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:58.827034 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:58.970075 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:58.970223 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:58.987653 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:58.998940 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:59.002468 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:59.023360 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
I20240502 14:32:59.023469 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:32:59.132273 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:32:59.188433 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:59.199354 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:32:59.205559 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:32:59.214119 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:59.223893 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:59.237902 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:59.280761 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 14:32:59.284678 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
I20240502 14:32:59.404971 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:59.407428 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:59.413355 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:59.420018 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:59.437582 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:32:59.518759 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:32:59.596844 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240502 14:32:59.615934 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:32:59.621979 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:59.635807 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:59.648191 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:59.666469 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:59.673740 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:32:59.740624 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:32:59.809136 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:32:59.836066 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:32:59.857434 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:32:59.872416 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:32:59.875689 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:32:59.879514 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:32:59.883312 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:00.029103 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:33:00.061519 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:00.116678 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:00.117169 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:00.128041 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:33:00.137168 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:00.140714 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:00.153884 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:00.173040 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240502 14:33:00.188333 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 14:33:00.255743 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:00.340397 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:00.343639 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:33:00.346117 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:00.356995 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:00.357456 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:00.359905 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 57 similar messages]
I20240502 14:33:00.371939 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:00.518200 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:00.579492 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:33:00.588492 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:00.596685 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:00.617879 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:00.625491 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:00.638985 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:00.659602 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:33:00.733580 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:33:00.806232 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:33:00.827420 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:00.833818 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:00.861667 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:00.862807 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:00.867648 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:00.889348 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:00.904754 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:33:01.004837 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:01.061547 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:01.068202 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:01.069787 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:01.081650 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:01.086797 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:33:01.092840 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:01.122475 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:33:01.125432 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:33:01.154526 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:33:01.231833 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:33:01.275529 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:01.277442 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:01.289705 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:01.306977 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:01.310730 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:01.338346 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240502 14:33:01.425454 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:01.497962 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:01.499215 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:01.502058 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:01.503158 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:01.505728 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:01.612177 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:01.644081 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:33:01.754529 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:33:01.758751 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:01.764266 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:01.764330 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:01.772303 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:01.778892 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:01.801352 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:33:01.808720 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240502 14:33:01.914943 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:01.968341 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:01.970219 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:01.981813 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:02.005609 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:02.013293 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:02.089151 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240502 14:33:02.152817 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:02.199489 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:02.214067 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:02.226089 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:02.234318 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:02.256590 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:02.281909 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:33:02.328706 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:33:02.329706 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:02.337204 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:33:02.392773 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:33:02.414611 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:33:02.423276 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:02.435233 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:02.445042 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:02.449520 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:02.462388 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:02.648269 18544 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240502 14:33:02.657802 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:02.665613 18544 catalog_manager.cc:4573] TS a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 96662a3d9e1b4f03b05464ddd06b93d6: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:02.676460 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:02.686179 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:02.693717 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:02.696784 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:02.713337 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:02.733172 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240502 14:33:02.742471 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:02.781857 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240502 14:33:02.794034 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:33:02.891727 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:02.899575 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:02.899868 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:02.902495 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:02.927318 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:02.970242 12794 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:33:02.970393 12794 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:33:02.970983 12794 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
W20240502 14:33:02.972440 18636 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:02.982409 18637 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:33:02.982543 18637 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: 7ab6d395b30d4afb9821b61c9454430b, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:33:02.982789 12794 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:33:03.014190 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:33:03.103071 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:33:03.137131 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:33:03.137780 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:03.150894 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:03.156414 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:03.171097 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:03.182776 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:03.205844 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:03.237983 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:33:03.313742 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:03.385733 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:03.396011 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:03.404922 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:03.420966 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:03.426651 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:03.455845 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:33:03.484767 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:33:03.596702 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:33:03.608067 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:03.627552 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:03.631233 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240502 14:33:03.632454 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:03.638051 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:03.641969 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:03.654848 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
I20240502 14:33:03.665105 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:03.673642 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:33:03.724875 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:33:03.871181 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:03.872649 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:03.881165 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:03.885831 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:03.886749 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:03.901237 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:03.932150 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:33:03.935232 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:33:04.006006 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:33:04.027408 18544 catalog_manager.cc:4573] TS a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6a992d12aee7416096d1353533397386: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:04.064393 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:04.089089 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:04.097235 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:04.098487 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:04.108395 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:04.115989 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:04.282301 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:04.299193 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:04.304317 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:04.325316 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:04.342280 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:04.343046 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
I20240502 14:33:04.343642 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:04.518002 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:04.518256 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:04.536437 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:04.550886 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:04.555591 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:04.582433 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:33:04.583402 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240502 14:33:04.668392 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:04.688740 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:33:04.729171 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:33:04.730528 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:33:04.761022 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:04.775398 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:04.781483 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:04.794503 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:33:04.797160 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:04.805632 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:04.842386 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:05.017211 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:05.035909 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:05.051733 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:05.058388 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:05.064343 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:05.135038 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240502 14:33:05.206703 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:05.237418 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:05.240007 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:05.249971 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:05.264329 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:05.272797 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:05.277216 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:05.290534 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240502 14:33:05.298022 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:33:05.437742 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:05.442148 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:05.455778 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:05.456305 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:05.475392 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:05.486347 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240502 14:33:05.535524 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:33:05.537895 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:33:05.541260 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:33:05.667799 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:05.674124 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:05.674872 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:05.692488 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:05.702972 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:05.705861 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:05.727182 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:05.763810 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:33:05.775394 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:33:05.904299 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:05.911788 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:05.923831 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:05.924649 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:05.951345 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:05.982769 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:33:05.987082 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:33:06.100958 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:06.107930 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
I20240502 14:33:06.152652 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:06.155130 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:06.180769 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:06.181089 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:06.181994 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:06.273552 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:06.313035 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:33:06.369781 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:33:06.388155 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:06.390030 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:06.392117 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:06.399042 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:06.401285 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:06.450037 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:33:06.471977 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:33:06.514166 18544 catalog_manager.cc:4573] TS a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 0ffb6e582ef04fb38dd273c13febac58: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:06.535573 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:06.544100 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:33:06.612138 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:06.613523 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:06.632414 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:06.640592 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:06.655140 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:06.764506 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240502 14:33:06.768380 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:06.832892 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:33:06.843585 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:06.846297 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:06.867492 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:06.868554 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:06.900622 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 14:33:06.902218 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:06.966806 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:33:07.057498 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:07.069119 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:07.077011 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:07.077560 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:07.079136 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:07.099203 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:07.274197 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:07.278617 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:07.288170 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:07.299288 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:07.315394 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:07.329536 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 14:33:07.330509 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:07.393929 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240502 14:33:07.431761 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:33:07.482738 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:07.496887 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:07.505841 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:07.513382 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:07.523110 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:07.528740 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:33:07.556300 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:07.702173 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
I20240502 14:33:07.710798 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:07.732054 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:07.732812 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:07.739842 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:07.746922 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:07.751076 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:33:07.855206 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:07.857808 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:33:07.956789 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:07.957376 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:07.960958 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:07.964774 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:07.969646 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:08.011359 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240502 14:33:08.036319 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:33:08.123148 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:08.168190 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:08.172679 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:08.181567 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:08.185406 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:08.199144 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:08.213965 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:33:08.240020 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:33:08.262522 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:08.282867 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:33:08.366425 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:08.371197 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:08.392195 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:33:08.394315 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:08.398106 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:08.419139 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:08.443711 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
I20240502 14:33:08.590488 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:08.593712 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:08.622180 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:08.628484 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:08.642930 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:08.655115 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:08.659767 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:33:08.672627 18768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240502 14:33:08.677989 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:08.736114 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:33:08.826519 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:08.843613 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:08.851264 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:08.854135 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:08.857345 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:08.899060 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:33:08.932801 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:33:09.006687 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:33:09.083530 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:09.097003 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:09.104389 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:09.109146 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:09.117447 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:09.138670 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:09.141152 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:33:09.273653 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:09.312973 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:09.318125 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:09.322806 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
I20240502 14:33:09.355464 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:09.357445 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:09.373682 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:09.411656 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:33:09.567965 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:09.581128 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:09.586635 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:09.587111 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:09.598111 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:09.620565 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:09.667199 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:33:09.696638 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:33:09.712291 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:09.712857 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:33:09.801694 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:09.809268 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:09.819285 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:09.820195 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:09.824676 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:09.955096 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
I20240502 14:33:10.022091 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:10.041749 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:10.048314 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:10.052125 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:10.055140 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:10.066612 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:10.093808 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:33:10.109979 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:10.162592 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:33:10.242452 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:10.246636 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:10.252841 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:10.253428 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:10.270587 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:10.311079 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240502 14:33:10.412925 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:33:10.454916 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:10.458321 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:10.478327 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:10.483318 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:10.495456 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:10.520397 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:10.535143 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 14:33:10.536101 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:33:10.604406 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:10.617061 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:33:10.617803 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240502 14:33:10.621306 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:33:10.655696 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:10.696705 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:10.697010 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:10.709174 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:10.725366 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:10.744504 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:33:10.782349 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:33:10.884755 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:10.894394 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:10.896839 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:10.909679 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:10.915896 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:10.951925 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:33:11.039780 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:11.044585 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:33:11.062466 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:11.096503 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
I20240502 14:33:11.129966 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:11.133512 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:11.154266 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:11.156811 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:11.159010 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:11.340440 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:33:11.347731 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:11.360481 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:11.370677 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:11.371343 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:11.384728 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:11.506448 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:11.510900 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:33:11.571691 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:33:11.579028 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:11.580989 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:11.598748 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:11.613168 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:11.625057 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:11.625723 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:11.628100 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:33:11.796165 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:11.807512 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:11.824499 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:11.826180 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:11.852679 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:11.897240 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:33:11.972811 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
I20240502 14:33:12.028126 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:12.052400 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:12.053759 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:12.062834 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:12.071143 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:12.081871 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:12.103664 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:33:12.147872 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:12.255602 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 14:33:12.263633 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:12.272804 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:12.285562 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:12.286224 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:12.302306 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:12.358805 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240502 14:33:12.393206 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 14:33:12.468832 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:12.472983 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:12.490924 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:12.492010 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:12.495220 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:12.645426 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:12.662300 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:33:12.680987 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240502 14:33:12.695276 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:12.720809 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:12.734328 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:12.740651 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:33:12.762730 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:12.763306 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:12.763768 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:12.862269 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:33:12.901754 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240502 14:33:12.916992 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:33:12.955129 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:12.975229 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:12.982214 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:12.983122 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:12.987782 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:13.052059 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:13.065953 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:33:13.140597 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:13.224043 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:13.226722 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:13.241284 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:13.250805 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:13.251842 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:13.261217 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:33:13.266831 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:33:13.296991 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:33:13.312534 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:33:13.432477 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:13.437464 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:13.451119 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:13.461032 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:13.463869 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:13.501121 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:33:13.501649 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:13.578433 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:33:13.634621 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
I20240502 14:33:13.694176 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:13.706182 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:13.707044 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:13.710208 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:13.711851 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:13.744799 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240502 14:33:13.758809 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:13.818177 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:33:13.847409 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:33:13.848510 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:13.897253 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:33:13.906903 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:13.914746 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:13.916570 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:13.917408 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:13.917515 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:14.125820 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:14.129369 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:14.132567 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:33:14.137307 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:14.146878 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:14.158694 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:14.287200 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:14.294647 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:14.353008 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:14.355639 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:14.369905 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:14.371136 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:14.374186 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:14.392839 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:33:14.499892 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
I20240502 14:33:14.565135 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:14.574790 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:14.582281 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:14.591061 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:14.608743 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:14.708874 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:14.810365 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:14.826983 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:14.837802 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:14.849262 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:33:14.857509 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:14.877374 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:14.907562 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:14.921440 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:33:14.927536 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:33:14.928364 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
I20240502 14:33:15.075014 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:15.078521 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:15.098264 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:15.116370 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:15.126278 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:15.133749 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:33:15.182317 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:15.294692 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:33:15.297298 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:15.302575 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:15.326840 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:15.327502 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:15.337355 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:15.347148 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240502 14:33:15.375416 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:15.473542 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:33:15.481374 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:33:15.512581 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:15.515703 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:15.527047 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:15.530812 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
I20240502 14:33:15.544030 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:15.546370 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:15.578078 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:15.636118 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 14:33:15.646579 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:33:15.680388 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:33:15.745009 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:15.745169 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:15.771036 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:15.799713 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:15.804967 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:15.828565 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:33:15.859001 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:33:15.877924 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:15.957408 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:15.981060 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:33:15.994323 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:15.996817 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:16.009157 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:16.013584 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:16.042924 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:16.066012 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240502 14:33:16.142576 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:33:16.210999 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:16.215262 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:16.225210 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:16.226215 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:16.229396 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:16.294565 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:33:16.383057 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:16.410530 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:16.419131 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:16.433581 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:16.436980 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:16.462448 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:16.487196 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:16.567078 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:33:16.627595 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:33:16.629468 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:16.663208 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:16.668428 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:33:16.675071 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:16.683028 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:16.709292 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:16.734848 16116 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:33:16.735090 16116 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:33:16.735807 16116 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
W20240502 14:33:16.738087 18636 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:16.752152 18637 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:33:16.752271 18637 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: 7ab6d395b30d4afb9821b61c9454430b, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:33:16.752667 16116 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:33:16.833302 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:16.900218 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:16.908594 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:16.910084 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:16.910754 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:16.918776 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:16.947867 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240502 14:33:16.962103 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:17.066025 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:33:17.099998 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:17.123543 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:17.130148 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:17.139066 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:17.169905 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:17.218982 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240502 14:33:17.223687 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:33:17.306684 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:33:17.336946 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:17.347311 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:33:17.355976 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:17.359696 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:17.362668 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 14:33:17.365497 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:17.374909 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:17.384892 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:17.460040 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:17.487093 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 14:33:17.556106 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:17.562875 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:17.567175 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:17.572742 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:17.603255 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:17.750226 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:17.785302 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:17.793841 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:17.803020 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
I20240502 14:33:17.803393 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:17.812098 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:33:17.813508 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:17.813983 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:33:17.833312 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:17.892649 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:33:17.924258 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240502 14:33:17.971482 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:17.988750 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:17.998215 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:18.012099 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:18.018579 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:33:18.023039 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:18.039793 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:18.190441 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:33:18.250780 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:18.262564 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:33:18.263476 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:33:18.278402 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:18.283898 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:18.284212 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:18.293287 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:18.311941 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:33:18.313747 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:18.463979 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:18.476586 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:18.480448 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:18.483573 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:18.497895 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:18.554386 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:33:18.579736 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:18.670011 16116 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:33:18.670193 16116 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:33:18.670950 16116 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
W20240502 14:33:18.673540 18636 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:18.674929 18637 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:33:18.675069 18637 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: 7ab6d395b30d4afb9821b61c9454430b, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:33:18.675534 16116 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:33:18.712647 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:18.715351 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:18.722014 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:18.727705 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:18.730398 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:18.735566 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:18.743279 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:33:18.804637 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240502 14:33:18.827790 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240502 14:33:18.880237 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:33:18.943326 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:18.946954 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:18.957315 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:18.984894 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:18.985913 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:19.021575 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:33:19.143414 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:19.156936 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:19.213112 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:19.213763 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:19.242076 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:19.245349 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:19.247395 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:19.394853 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:33:19.443408 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:19.461545 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:19.462059 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:19.476590 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:19.486771 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:33:19.500876 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:19.552268 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:19.660303 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
I20240502 14:33:19.702320 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:19.702488 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:19.702577 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:19.706032 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:19.706506 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:19.715448 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:19.726779 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:33:19.743042 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240502 14:33:19.834380 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:33:19.880295 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:19.893334 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:19.903702 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:19.905249 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:19.908993 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:19.909922 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:33:20.008231 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:20.105728 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:20.106803 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:33:20.115345 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:20.130666 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 14:33:20.131871 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:20.133078 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:20.153301 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:20.275033 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:20.334291 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:20.336920 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:20.341255 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:20.347916 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:20.353646 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:20.408645 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:33:20.448235 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:20.573427 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
I20240502 14:33:20.574858 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:20.578157 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:20.582744 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:20.583738 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240502 14:33:20.602998 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:33:20.604429 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:33:20.619076 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:20.628307 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:20.726408 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:33:20.738471 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240502 14:33:20.739261 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:33:20.743790 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:20.820632 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:20.822052 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:20.838660 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:20.849073 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:20.866494 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:20.872285 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:20.951668 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:33:20.986414 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:33:21.040678 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:33:21.055840 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:21.063349 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:21.086333 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:21.088994 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:21.093709 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:21.165059 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:33:21.264047 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:21.269506 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:21.278846 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:21.280797 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:21.306102 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:21.308609 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:21.310691 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:21.453219 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
I20240502 14:33:21.469533 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:21.474334 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:21.487140 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:21.489778 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:21.491262 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:21.608443 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:33:21.640179 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:33:21.708673 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:21.719182 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:21.730314 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:21.745815 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:21.747277 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:21.844086 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:21.868870 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:21.910068 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:21.917536 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:21.928161 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:21.931080 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:21.941221 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:22.028874 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:33:22.121393 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:33:22.159631 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
I20240502 14:33:22.166148 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:22.175118 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:22.191989 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:22.194519 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
I20240502 14:33:22.196961 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:22.225042 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:33:22.227020 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:22.230361 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:22.384737 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:22.404749 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:22.415623 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:22.427265 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:22.435164 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:22.436316 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:22.444686 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:33:22.568603 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:33:22.628494 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:33:22.637656 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:22.660734 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:22.661427 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:33:22.664467 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:33:22.668874 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:33:22.678805 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:22.699013 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:22.703888 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:22.740471 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:22.795934 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
I20240502 14:33:22.880657 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:22.891001 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:22.900512 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:22.909338 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:22.911824 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:22.919279 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:33:22.919481 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:23.024050 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
I20240502 14:33:23.085777 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:23.101254 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:23.105480 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:23.121699 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:23.139011 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:23.141911 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:23.236204 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:33:23.305281 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:23.320506 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:33:23.326996 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:23.330109 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:23.334117 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:23.338068 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:23.354007 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:23.473728 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:33:23.530732 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:23.537374 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:23.537690 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:23.538372 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:23.540705 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:23.564177 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:33:23.606390 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:23.632597 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:33:23.658062 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:33:23.735831 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:23.752913 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:33:23.776042 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:23.789844 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:23.793541 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:23.803411 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:23.816242 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:23.844913 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240502 14:33:23.974437 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:33:23.984251 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:24.001927 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:24.008219 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:24.033291 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:24.043767 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
I20240502 14:33:24.047363 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:24.060027 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:24.168900 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:24.230028 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:24.245143 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:24.258822 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:24.265534 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:24.293128 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:24.313850 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:33:24.470666 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:24.472920 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:24.476332 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:24.477857 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:24.498622 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:24.521656 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:24.532969 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240502 14:33:24.534765 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:33:24.616513 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:33:24.638667 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:24.685392 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:24.700168 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:24.700719 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:33:24.706899 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:24.722378 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:24.743546 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:24.906800 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:33:24.915313 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:24.921998 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:24.925736 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:24.928922 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:24.929283 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:24.936487 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:33:25.034802 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:33:25.075968 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:25.086015 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:25.086056 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
I20240502 14:33:25.118589 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:25.128664 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:25.144111 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:25.144119 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:25.145277 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:25.148138 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:33:25.149957 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:33:25.161480 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
I20240502 14:33:25.371935 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:25.382267 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:25.382265 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:25.387490 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:25.397042 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:25.523303 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:33:25.536195 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
I20240502 14:33:25.629890 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:25.631736 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:25.632534 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:25.637135 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:33:25.663269 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:25.677680 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:25.680208 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:25.698480 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:25.751986 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 14:33:25.827193 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:33:25.862419 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:25.864112 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:25.869864 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:25.874295 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:25.879354 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:25.915529 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:33:26.076606 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:26.081835 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:26.089299 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:26.099788 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:26.113734 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:26.114013 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:26.133481 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:26.221230 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:33:26.274410 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:33:26.290522 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:33:26.298223 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:26.301672 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:26.308311 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:26.312366 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:26.344015 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:26.377059 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:33:26.460467 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240502 14:33:26.496891 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:33:26.509649 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:26.516026 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:26.523264 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:26.525355 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:26.538820 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:26.623581 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:26.629132 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:26.741768 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:26.760638 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:26.761755 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:26.767603 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:26.783195 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:26.783797 17962 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:33:26.783969 17962 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:33:26.784610 17962 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
W20240502 14:33:26.794493 18637 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:33:26.819783 18636 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:26.820155 18636 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: 7ab6d395b30d4afb9821b61c9454430b, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:33:26.820528 17962 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:33:26.844522 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240502 14:33:26.974581 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:33:26.982704 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:33:26.985021 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:26.991602 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:26.996016 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:27.001813 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:27.020534 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:33:27.024904 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:27.126833 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:33:27.196261 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:27.200202 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:27.213565 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:27.213981 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:27.230852 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:27.246009 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:27.251328 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:27.267472 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:33:27.415023 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:33:27.448976 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:27.464545 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:27.469904 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:27.470470 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:27.486482 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:27.574923 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:33:27.653298 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:33:27.667872 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:27.689582 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:27.693778 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:27.701210 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:27.703357 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
I20240502 14:33:27.721410 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:27.743398 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:27.809028 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:33:27.810019 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:27.831048 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:33:27.838313 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 14:33:27.898348 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:27.905596 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
I20240502 14:33:27.913633 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:27.914513 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:27.923054 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:27.936463 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:27.942574 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240502 14:33:28.012686 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:33:28.104876 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:28.116340 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:28.134654 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:28.138127 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:28.138713 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:28.267851 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:28.300146 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:33:28.342999 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:28.362489 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:28.364919 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:28.365162 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:28.365800 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:28.366472 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:28.386672 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:33:28.542133 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:28.550129 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:33:28.554414 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:28.582275 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:28.586311 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:28.586545 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:28.747212 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:33:28.796416 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:28.796960 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:28.805377 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:28.810811 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:28.814224 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:28.827474 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:28.889806 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240502 14:33:28.891031 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:28.958401 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:33:28.982786 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:33:28.988415 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
I20240502 14:33:29.011081 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:29.024951 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:29.029709 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:29.037636 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:29.037834 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:29.138552 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240502 14:33:29.154670 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:33:29.229583 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:29.237900 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:29.244750 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:29.249467 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:29.250392 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:29.411103 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:29.419539 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:33:29.441289 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:29.445389 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:29.445459 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:29.449673 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:29.461957 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:29.492421 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:29.534751 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:33:29.645058 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:33:29.677618 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:29.714152 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:29.725244 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:29.732439 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:29.738600 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:29.829686 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:33:29.892814 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:33:29.938736 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:29.940214 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:29.953275 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:29.955092 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:29.963588 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:29.965062 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:29.969388 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:30.046244 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:33:30.092638 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:33:30.181948 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:33:30.183974 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:30.189543 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:30.202550 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:30.204777 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:30.218866 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:30.271502 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 14:33:30.277789 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240502 14:33:30.290827 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:33:30.363482 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:33:30.369738 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:30.375197 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:30.403020 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:30.404917 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:30.419922 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:30.422945 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:30.428393 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:30.579207 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
I20240502 14:33:30.631085 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:30.634598 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:30.639075 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
I20240502 14:33:30.648574 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:30.648677 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:30.660181 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:30.755247 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 14:33:30.785916 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:33:30.816311 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:30.855499 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:33:30.867130 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:30.871752 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:30.876027 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:30.876873 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:30.892012 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:30.917220 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:31.038089 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:33:31.075057 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:31.080765 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:31.087698 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:31.091146 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:31.107244 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:33:31.110183 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:31.186398 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
I20240502 14:33:31.311025 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:31.312227 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:31.319087 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:31.330961 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:31.341135 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:31.348664 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:31.362207 18410 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:33:31.362388 18410 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:33:31.363045 18410 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
W20240502 14:33:31.364794 18636 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:31.376613 18637 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:33:31.376724 18637 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: 7ab6d395b30d4afb9821b61c9454430b, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:33:31.376940 18410 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:33:31.492986 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:31.555493 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:31.563336 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:31.564824 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:31.569139 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:31.569967 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:31.572134 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:33:31.751145 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240502 14:33:31.756213 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:33:31.784149 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:31.795574 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:31.796382 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:31.799821 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:31.808573 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:31.873279 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:31.905009 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:33:31.939560 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:31.961923 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:33:32.038954 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:32.039060 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:32.048941 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:32.053438 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:32.063452 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:32.204897 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:33:32.226487 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:33:32.253727 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:32.258934 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:32.261883 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:32.269631 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:32.277168 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:32.277508 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:32.294878 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:33:32.355165 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:33:32.367327 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:33:32.485324 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:32.490047 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:32.490466 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:32.492810 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:32.495556 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:32.542690 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:33:32.544100 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:32.550483 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240502 14:33:32.679940 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:33:32.712433 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:32.719696 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:32.721642 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:32.729454 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:32.745723 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:32.808640 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:32.877223 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:33:32.892285 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:33:32.893365 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:33:32.950256 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:32.960693 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:32.971303 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:32.974391 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:32.991425 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:33.053282 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:33:33.054495 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:33.183754 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:33.202093 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:33.216724 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:33.219084 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:33.233887 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:33:33.234172 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
I20240502 14:33:33.242338 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:33.256839 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240502 14:33:33.345719 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:33.405946 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:33.409359 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:33.412039 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:33.413657 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:33.425232 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:33.642788 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:33.650652 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:33.651912 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:33.671687 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:33.673264 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:33:33.684794 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
I20240502 14:33:33.693495 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:33.705825 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:33.746971 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:33:33.879293 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:33.890451 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:33.892752 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:33.905473 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:33:33.907711 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:33.918924 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:33.919250 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240502 14:33:33.920316 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:34.054574 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:33:34.071231 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:33:34.116777 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:34.117115 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:34.127022 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:34.129802 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:34.129930 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:34.182443 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240502 14:33:34.256057 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:33:34.269728 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:34.325917 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:34.339479 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:34.353916 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:34.360913 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:34.369514 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:34.430073 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:34.523483 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:33:34.531121 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:34.541301 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:34.549597 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:34.557407 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:34.565412 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:34.652984 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:33:34.755848 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:34.764525 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:34.777617 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:34.786208 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:34.787310 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:34.806766 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:34.807312 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:33:34.905288 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:33:34.954234 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:34.960018 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:33:35.022642 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:33:35.039382 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:35.048198 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:35.055054 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:35.057015 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:35.080005 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:35.082343 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:33:35.085724 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:33:35.095137 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:33:35.249382 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
I20240502 14:33:35.268281 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:35.279276 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:35.279990 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:35.281107 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:35.290719 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:35.304325 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:35.338338 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:35.463968 18544 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 14:33:35.470741 18544 catalog_manager.cc:4573] TS a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 96662a3d9e1b4f03b05464ddd06b93d6: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:35.502156 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:35.509145 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:35.512728 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:35.514211 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:35.516402 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:35.530112 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:33:35.565586 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240502 14:33:35.611001 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:33:35.629199 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:33:35.697763 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
I20240502 14:33:35.753587 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:35.763275 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:35.763885 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:35.764549 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:35.768679 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:35.809762 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 14:33:35.829732 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:35.866106 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:35.989506 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:35.990252 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:36.009457 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:36.017428 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:36.022629 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:36.114233 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:33:36.151410 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
I20240502 14:33:36.248386 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:36.252841 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:36.257331 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:36.261629 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:36.263341 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:36.310024 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:33:36.355146 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:33:36.383939 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:33:36.384922 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:36.432597 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:36.439612 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:33:36.471709 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:36.475940 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:36.492746 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:36.493983 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:36.509299 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:36.635345 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240502 14:33:36.706981 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:36.713784 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:33:36.721784 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:36.723949 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:36.724027 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:36.748643 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:36.760658 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:33:36.786634 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:36.848451 18544 catalog_manager.cc:4573] TS a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6a992d12aee7416096d1353533397386: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:36.902890 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:36.916910 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:36.929260 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:36.930451 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:36.950408 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:36.962090 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:37.116644 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:33:37.117895 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:37.130311 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:37.151087 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:37.152035 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:37.181084 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:37.249117 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:33:37.350258 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:37.356691 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:37.357381 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:37.370592 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:37.372193 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:37.383654 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:37.409171 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:33:37.436024 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:33:37.446527 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:37.556730 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:33:37.559046 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:37.586822 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:37.588176 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:37.588521 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:37.600593 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:37.616608 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:33:37.655689 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:33:37.681478 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:33:37.889148 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:37.894124 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:37.894336 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:37.894724 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:37.901464 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:37.942917 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240502 14:33:37.969331 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:37.978037 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:33:37.999678 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:38.061168 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:33:38.120188 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:33:38.132341 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240502 14:33:38.150835 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:38.170804 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:38.177996 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:38.181617 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:38.208508 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:38.255820 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:33:38.281572 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240502 14:33:38.349330 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:38.355904 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:33:38.386186 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:38.388376 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:38.392304 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:38.392381 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:38.398651 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:38.417392 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:38.577926 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240502 14:33:38.583274 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:38.600879 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:38.626900 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:38.638937 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:38.658843 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:38.869838 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:38.871923 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:38.872740 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:38.885838 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:38.886418 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:38.920001 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:38.923130 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 97 similar messages]
W20240502 14:33:38.923803 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:33:38.924810 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:39.016031 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:33:39.065563 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:39.074843 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:39.081449 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:39.082073 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:39.087935 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:39.097833 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:33:39.098625 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:33:39.175958 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:33:39.227260 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:33:39.250906 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
I20240502 14:33:39.280079 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:39.292481 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:39.294327 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:39.295612 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:39.312197 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:39.321802 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:39.323856 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:33:39.344746 18544 catalog_manager.cc:4573] TS a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 0ffb6e582ef04fb38dd273c13febac58: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:39.365705 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:39.488946 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:39.500342 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:39.511200 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:39.523438 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:39.523967 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:39.714699 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:39.759596 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:39.761765 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:39.762212 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:39.774451 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:39.787084 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:33:39.837831 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:39.847563 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:39.892418 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:33:39.956090 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:33:40.006819 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:40.019511 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:33:40.030754 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:40.031374 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:40.053503 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:40.054445 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:40.075062 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:33:40.187407 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:40.230909 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:40.232546 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:40.236871 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:40.242954 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:40.244068 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:40.326231 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:33:40.377568 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:33:40.382493 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:40.422581 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
I20240502 14:33:40.445051 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:40.445717 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:40.446969 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:40.451333 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:40.462255 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:40.479276 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:33:40.552979 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:33:40.586953 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:40.618429 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
I20240502 14:33:40.686618 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:40.686851 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:40.696825 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:40.708046 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:40.726773 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:33:40.728327 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240502 14:33:40.738205 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:40.815729 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 14:33:40.823479 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:33:40.834182 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:40.895390 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:40.904543 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:40.926002 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:40.931164 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:40.935607 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:40.955263 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240502 14:33:40.997627 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:41.122294 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:41.140659 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:41.146796 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:41.147234 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:41.150528 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:41.348538 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:41.356211 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:41.365284 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:41.372572 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:41.374539 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:33:41.378820 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:41.387769 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:41.390017 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:41.430331 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:33:41.461247 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:33:41.584154 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:33:41.590255 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:41.612614 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:41.623164 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:41.623868 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:41.655926 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:41.713758 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:33:41.733711 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:33:41.773626 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:41.816570 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:41.843557 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:41.857127 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:41.858163 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:41.865658 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:41.872869 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:41.971714 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:33:42.048822 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
I20240502 14:33:42.063956 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:42.077579 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:42.084408 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:42.089438 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:42.110790 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:42.153273 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:42.209538 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:42.219354 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:33:42.273600 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:42.283339 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:42.287950 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:42.293697 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:42.300971 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:42.460377 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:33:42.470925 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:33:42.498896 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:33:42.508805 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:42.512989 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:42.513451 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:42.516139 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:42.526898 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:33:42.529924 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:42.551101 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:33:42.627275 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:42.707612 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:42.743422 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:42.756561 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:42.763233 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:42.764304 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:42.781981 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:33:42.782963 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:42.800987 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240502 14:33:42.938532 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:33:43.017505 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:43.020731 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:43.025449 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:43.029937 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:43.033416 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:43.062546 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:43.087560 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:33:43.107223 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240502 14:33:43.112955 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240502 14:33:43.190866 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:43.212631 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:43.217147 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:43.218508 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:43.226277 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:43.239246 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:43.435400 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 14:33:43.440420 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:43.448853 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:43.459095 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:43.474835 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:43.478219 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:43.509831 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240502 14:33:43.513414 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:33:43.670238 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:43.670929 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:43.685734 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
I20240502 14:33:43.691445 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:43.704409 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:43.713721 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:43.727685 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:43.730788 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:43.785463 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:33:43.907085 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:43.914916 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:33:43.922585 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:43.930673 18768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 103 similar messages]
I20240502 14:33:43.932476 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:43.943475 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:43.952054 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:44.046972 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:33:44.105299 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:44.131076 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:33:44.140561 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:44.159091 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:44.184793 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:44.185135 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:44.192659 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:44.230798 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:33:44.230937 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:44.292532 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:33:44.388265 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:44.393664 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:44.402271 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:44.413652 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:44.414317 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:44.424523 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240502 14:33:44.485491 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:33:44.593535 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:44.629937 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:44.640658 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:44.653539 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:44.656836 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:33:44.669646 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:44.676751 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:44.695639 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:44.856402 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:33:44.874368 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:44.886294 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:44.887846 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:44.889297 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:44.909945 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:44.944336 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:33:45.076985 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:33:45.085826 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:45.098809 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:45.112090 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:45.114391 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:45.123308 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:33:45.125000 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:45.127384 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:45.133471 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:45.259588 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:33:45.311172 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
I20240502 14:33:45.335122 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:45.336424 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:45.337430 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:33:45.340453 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:45.346618 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:45.353231 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:45.400804 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240502 14:33:45.404309 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240502 14:33:45.503453 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:33:45.530926 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:45.532359 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:45.546627 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:45.547166 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:45.557540 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:45.565390 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:33:45.571605 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:45.618312 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:45.775638 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:45.779141 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:45.786388 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:45.786718 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:45.818300 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:45.833465 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 14:33:45.888631 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:33:46.010923 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:46.037416 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:46.043519 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:46.048657 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:46.050953 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:46.054661 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:46.084100 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:33:46.114245 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:33:46.115445 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:46.195571 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
I20240502 14:33:46.293603 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:46.299932 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:46.300906 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:46.306056 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:46.306140 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:46.417538 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:46.506932 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:46.509688 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:46.523685 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:46.523844 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:46.526535 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:46.539306 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:33:46.549080 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:33:46.652524 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:33:46.671391 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:33:46.671700 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:46.728518 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:46.732084 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:46.738124 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:46.752135 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:46.758095 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:46.803117 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:46.899024 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:33:46.938530 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:46.943416 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:46.950691 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:46.960705 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:46.966032 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
I20240502 14:33:46.972777 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:47.048099 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:47.065325 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:33:47.157788 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:47.158303 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:47.159549 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:47.171829 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:33:47.175014 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:47.180936 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:47.252300 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:33:47.281653 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:33:47.366917 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:47.384769 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:47.391875 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:47.392958 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:47.393370 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:47.405948 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:47.434307 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:33:47.475581 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:47.502843 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:33:47.521272 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:33:47.600625 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
I20240502 14:33:47.601831 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:47.621223 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:47.628388 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:47.636791 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:47.654942 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:47.836912 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:47.846110 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:47.855464 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:47.870015 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:47.870177 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:47.887897 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240502 14:33:47.891304 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:33:47.897534 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:33:47.916546 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:33:47.943403 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240502 14:33:47.978120 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:47.995522 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:48.101910 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:48.107223 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:48.119333 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:48.116864 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:48.123603 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:48.353950 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:48.354697 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:48.383142 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:48.383776 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:48.400456 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:48.457993 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:33:48.532387 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:48.555292 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:33:48.571291 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:48.573083 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:48.597576 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:48.606828 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:48.624248 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:48.640822 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:48.642606 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:33:48.748077 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240502 14:33:48.805433 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:33:48.844883 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:48.859230 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:48.861743 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:48.878342 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:48.906389 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:33:48.913457 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:48.932770 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240502 14:33:48.958571 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:48.979977 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:33:49.024446 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:33:49.060683 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:49.086650 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:49.087291 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:49.090432 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:49.093834 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:49.102371 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:49.199108 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:33:49.317842 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
I20240502 14:33:49.334690 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:49.336516 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:49.348256 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:49.358829 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:49.370012 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:49.445045 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:33:49.492250 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:49.531832 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:49.536619 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:49.537451 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:49.550163 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:49.557425 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:49.557829 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:49.670526 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:33:49.743563 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:33:49.748394 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:49.749498 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:49.772508 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:49.773344 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:49.775986 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:33:49.777922 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:49.831496 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:33:49.870400 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:33:49.879451 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:33:49.976897 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:33:49.997131 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:50.002650 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:50.009228 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:50.009245 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:50.013022 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:50.018599 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:50.038343 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:50.169263 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:33:50.224398 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:50.235293 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:50.239264 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:50.243275 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:50.257982 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:50.348390 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240502 14:33:50.358646 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240502 14:33:50.410265 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
I20240502 14:33:50.444355 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:50.446031 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:50.452617 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:50.461920 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:50.466266 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:50.471215 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:33:50.473789 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:50.476318 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:50.575428 22661 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:33:50.575626 22661 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
W20240502 14:33:50.575848 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:33:50.576383 22661 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
W20240502 14:33:50.577826 18636 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:50.578599 18637 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:33:50.578729 18637 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: 7ab6d395b30d4afb9821b61c9454430b, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:33:50.578986 22661 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:33:50.668169 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:50.681654 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:50.708055 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:50.709332 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:50.730067 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:50.874341 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:50.902467 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240502 14:33:50.945832 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:50.946190 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:50.949453 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:50.951364 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:50.958303 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:51.046248 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:51.146531 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:33:51.158075 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:51.174825 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:51.177392 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:51.183873 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:51.223475 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:51.252337 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 14:33:51.276089 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:33:51.303172 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:33:51.363946 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:51.377414 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:33:51.384020 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:51.389379 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:51.405728 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:51.406093 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:51.444900 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:51.455040 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:33:51.468367 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:51.481446 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
I20240502 14:33:51.586973 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:51.604249 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:51.633523 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:51.658120 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:51.675228 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:51.794359 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:33:51.836438 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240502 14:33:51.877513 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:51.890642 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:51.894778 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:51.897578 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:51.901613 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:51.925463 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:33:51.926564 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:51.976261 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:51.987113 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:33:52.095603 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:33:52.119031 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:52.124855 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:52.127394 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:52.129217 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:52.154407 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:52.157922 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:33:52.344413 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:52.357715 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:52.359238 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:52.366632 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:52.368068 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:52.429441 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:33:52.429605 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:33:52.447579 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:33:52.451889 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:33:52.454905 18766 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.65:36749, user_credentials={real_user=slave}} blocked reactor thread for 49731.2us
W20240502 14:33:52.457401 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:52.495774 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:52.634322 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:52.637508 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:52.638522 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:52.673626 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:52.685623 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:52.708246 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:33:52.826174 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240502 14:33:52.879067 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
I20240502 14:33:52.888290 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:52.892197 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:52.904533 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:52.907045 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:52.913530 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:52.959596 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:53.028501 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240502 14:33:53.061777 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:53.114953 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:53.123134 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:53.138554 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:53.143684 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:53.176301 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:33:53.186372 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:53.251493 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:33:53.324654 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:53.338096 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:53.340363 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:53.353289 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:53.368942 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:53.462394 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:53.570724 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 14:33:53.587950 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:53.609089 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:53.626391 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:53.631117 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:53.635398 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:53.648262 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:33:53.657281 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:53.684489 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:33:53.687803 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:33:53.844974 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:53.845938 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:53.859805 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:53.862432 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:53.869803 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:53.869813 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:53.887909 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:33:53.985625 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240502 14:33:53.986478 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:33:54.043088 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:33:54.076277 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:54.079366 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:54.081301 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:54.085716 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:54.089815 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:54.101783 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:54.159837 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240502 14:33:54.304754 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:33:54.326246 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:33:54.379271 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:54.382830 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:54.386312 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:54.394675 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:54.404112 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:54.410051 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:54.466277 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:33:54.515362 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:54.528048 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:33:54.588440 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:54.595300 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:54.615756 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:54.618351 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:54.623817 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:54.657986 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:33:54.730592 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240502 14:33:54.758723 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:33:54.804922 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:54.860558 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:54.861826 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:54.872860 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:54.876420 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:54.881598 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:54.953763 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:54.975287 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:33:55.074570 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:55.085366 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:55.112207 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:55.126468 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:55.128351 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:55.146682 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:33:55.153033 18900 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.68:41463, user_credentials={real_user=slave}} blocked reactor thread for 95550.4us
W20240502 14:33:55.180399 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:55.182668 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:33:55.187657 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:33:55.318898 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:55.324095 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:55.340744 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:55.341838 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:55.352592 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:55.420380 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
I20240502 14:33:55.543084 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:55.550757 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:55.558591 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:55.566030 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:33:55.570628 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:55.577103 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:55.583173 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:55.660478 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240502 14:33:55.675020 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240502 14:33:55.681979 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:55.791788 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:55.797469 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:55.799401 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:55.799850 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:55.821676 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:55.893687 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:33:55.982482 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 14:33:56.005781 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:56.018282 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:56.027169 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:56.029357 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:56.030350 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:56.033294 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:56.157164 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:33:56.166121 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:56.231662 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:56.234161 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:56.244947 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:56.249856 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:56.274343 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:56.355059 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:33:56.355862 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:33:56.450703 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:56.462119 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:56.465667 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:56.476775 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:56.477317 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:56.484688 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:56.523476 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:33:56.657552 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:56.682909 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:33:56.684875 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:56.688884 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
I20240502 14:33:56.704859 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:56.706130 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:56.716565 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:56.717916 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:56.763149 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:33:56.816534 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:33:56.872159 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:33:56.902386 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:56.905162 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:56.905850 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:56.911263 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:56.917423 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:56.919421 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:57.069531 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:33:57.075215 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:33:57.097295 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:33:57.097935 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:33:57.120812 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:57.134366 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:57.145794 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:57.156571 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:57.157747 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:57.197193 18769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.69:41119, user_credentials={real_user=slave}} blocked reactor thread for 47935.8us
W20240502 14:33:57.268340 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:57.268965 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
I20240502 14:33:57.355924 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:57.357681 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:57.359123 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:57.374406 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:57.376924 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:57.421579 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:57.580299 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:33:57.597431 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:33:57.611248 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:33:57.637507 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:57.640478 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:57.691931 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:57.693394 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:57.704264 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:57.854966 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:57.881044 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:57.884807 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:57.888079 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:57.891922 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:57.898808 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:57.914361 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:57.934888 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240502 14:33:57.961555 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240502 14:33:57.982714 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:33:58.073913 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
I20240502 14:33:58.094697 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:58.103055 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:58.111831 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:58.122212 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:58.123181 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:58.263183 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:58.273871 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:33:58.306560 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:58.312012 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:58.321599 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:58.322614 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:58.337750 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:33:58.340873 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:58.392633 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:58.406306 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:33:58.539876 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:58.541952 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:58.558065 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:58.563745 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:58.594714 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:58.635126 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:33:58.661761 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:33:58.723997 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:33:58.759032 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:33:58.787087 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:58.811949 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:33:58.813814 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:58.821430 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:58.833510 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:58.834246 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:58.845501 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:59.025868 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:59.035439 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 100 similar messages]
I20240502 14:33:59.047818 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:59.049145 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:59.049811 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:59.053683 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:33:59.071398 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:33:59.130035 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240502 14:33:59.137176 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:59.275784 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:59.276770 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:59.285077 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:33:59.288455 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:59.289412 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:59.297909 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:59.308750 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:33:59.347601 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:33:59.406301 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:33:59.473449 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:59.481263 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:59.484266 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:59.487475 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:59.488063 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:33:59.560686 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:33:59.610353 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:33:59.626951 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:33:59.705276 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:33:59.729058 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:33:59.736091 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:33:59.738472 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:59.743387 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:59.750319 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:59.765049 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:33:59.809461 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:33:59.865842 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
I20240502 14:33:59.936677 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:33:59.940578 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:33:59.958547 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:33:59.959919 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:33:59.989764 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:00.154789 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:00.170980 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:00.177714 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:00.198982 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:00.199641 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:00.204103 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:34:00.212121 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:00.232849 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:34:00.240661 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:34:00.252946 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240502 14:34:00.377151 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:00.400188 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:00.412482 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:00.419924 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:00.424496 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:00.437791 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:00.531811 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:34:00.536015 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
I20240502 14:34:00.644495 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:34:00.645562 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
I20240502 14:34:00.646245 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:00.676596 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:00.725778 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:34:00.725939 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:00.728384 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:00.729144 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:00.729770 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:34:00.793066 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:00.852010 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:34:00.903713 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:00.913587 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:00.921694 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:00.925307 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:34:00.932266 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:00.933708 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:01.115128 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:01.118491 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 14:34:01.122680 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:01.138964 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:01.139623 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:01.144183 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:01.158252 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:01.256556 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:01.335873 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:01.339082 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:01.373435 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:01.373886 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:01.392006 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:34:01.402097 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:34:01.402755 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:01.480242 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:34:01.553534 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:01.562619 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:01.565527 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:01.569574 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:01.576150 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:01.589078 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:01.658605 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:01.786655 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:01.798590 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:34:01.814397 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:01.818805 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:01.822117 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:01.829324 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:01.936933 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:34:01.998885 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240502 14:34:02.017452 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:34:02.027755 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:34:02.028219 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:02.030045 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:02.035367 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:02.041713 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:02.044093 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:02.066897 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:02.085060 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:34:02.190550 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:34:02.197417 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:02.251350 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:02.254892 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:02.266731 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:02.273494 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:02.275123 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:02.406188 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240502 14:34:02.431975 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:34:02.473299 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:02.477345 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:02.505332 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:02.512144 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:02.522624 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:02.531509 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:02.604671 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:02.686210 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:34:02.721412 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
I20240502 14:34:02.729684 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:02.738363 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:02.749570 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:02.765331 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:02.777623 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:02.815955 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240502 14:34:02.901631 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:34:02.961967 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:34:02.962986 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:02.978482 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:02.978533 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:02.981675 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:03.013481 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:03.013795 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240502 14:34:03.034970 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:34:03.047619 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:03.053135 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:03.168853 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:34:03.205750 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:03.219352 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:03.236001 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:03.237357 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:03.240123 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:03.262688 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:34:03.314944 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:34:03.410770 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:03.419823 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:03.422907 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:34:03.448809 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:34:03.450063 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:03.459270 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:03.491410 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:03.629931 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:03.633708 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:03.649098 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:03.657346 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:34:03.658558 24970 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:34:03.658695 24970 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:34:03.659332 24970 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
W20240502 14:34:03.660378 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:03.661034 18636 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:03.662787 18637 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:34:03.662894 18637 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: 7ab6d395b30d4afb9821b61c9454430b, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:34:03.663296 24970 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:34:03.683107 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:03.686697 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:03.791936 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 14:34:03.880954 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:03.884462 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:03.884514 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:03.890187 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:03.896983 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 14:34:03.905443 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:03.936185 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:34:04.069577 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:04.074297 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:04.085707 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 102 similar messages]
I20240502 14:34:04.098156 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:04.099812 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:04.101675 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:04.102006 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:04.198805 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:34:04.275926 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:04.285771 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:04.288975 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:04.296994 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:04.301885 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:04.328382 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:34:04.331271 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:04.369165 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240502 14:34:04.379365 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:34:04.381920 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:34:04.479809 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:34:04.505508 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:04.512015 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:04.512805 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:04.514081 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:04.520596 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:04.703301 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:04.706549 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:04.734855 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:04.737110 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:04.754850 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:04.770896 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:04.787912 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:04.828887 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:34:04.875385 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:34:04.948223 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
I20240502 14:34:04.953528 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:04.965229 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:04.967450 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:04.975744 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:05.036135 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:05.135355 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:34:05.153661 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:05.155421 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:34:05.162173 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240502 14:34:05.217890 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:05.223399 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:05.246065 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:05.248059 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:05.248466 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:05.260154 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:05.448160 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:05.451815 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:34:05.453647 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:34:05.464306 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:05.477412 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:05.482136 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:05.551235 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:34:05.556022 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:34:05.588884 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240502 14:34:05.649569 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:34:05.674235 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:34:05.685019 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:05.712855 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:05.716006 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:05.727147 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:05.745414 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:05.749760 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:05.753679 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240502 14:34:05.835671 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:05.931784 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:05.932658 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:34:05.942052 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:34:05.946451 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:05.961825 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:34:05.967505 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:05.968551 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:06.159235 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:06.160629 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:06.165614 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 64 similar messages]
I20240502 14:34:06.171342 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:06.173434 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:06.183986 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:06.186094 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:06.377652 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:06.389636 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:06.390429 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:06.406831 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:06.412592 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:06.437561 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:06.477440 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:34:06.512326 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 14:34:06.619906 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:06.625564 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:06.640705 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:06.641366 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:06.666707 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:34:06.673907 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:06.678258 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240502 14:34:06.695158 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:06.722676 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:34:06.769230 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:34:06.844074 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:06.857805 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:34:06.859171 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:34:06.861387 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:06.882182 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:06.890085 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:06.899041 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:07.055958 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:34:07.119027 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:07.130827 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:07.134840 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:07.137701 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:07.154045 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:07.188647 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:34:07.243899 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240502 14:34:07.252194 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:34:07.264094 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:07.301718 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:07.350075 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:07.373332 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:07.377745 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:07.381894 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:07.412936 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:07.446218 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:34:07.589233 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:07.591214 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:07.591950 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:07.601997 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:07.628584 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:07.630944 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240502 14:34:07.682085 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:07.730777 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:07.822367 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:07.825117 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:07.847533 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:34:07.856328 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:07.857106 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:34:07.868732 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:07.875427 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:07.896996 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:34:07.937477 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:34:07.958830 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
I20240502 14:34:08.066574 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:08.067531 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:08.072590 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:08.076941 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:08.102195 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:08.110234 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:08.161629 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:34:08.170454 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:08.252282 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:34:08.296510 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:08.305596 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:08.306190 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:08.337978 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:08.338390 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:08.430294 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:34:08.450309 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
I20240502 14:34:08.517517 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:08.520781 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:08.520890 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:08.543318 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:08.549026 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:08.551319 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:08.631342 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:34:08.773039 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:08.784835 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:08.801808 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:08.802588 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:08.803822 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:08.815665 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 14:34:08.892596 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:09.036199 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:34:09.045524 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:09.057313 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:09.068277 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:09.071203 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:09.072881 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:09.099941 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240502 14:34:09.106055 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:09.177855 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:34:09.276329 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:34:09.290875 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:09.294468 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:09.297235 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240502 14:34:09.306474 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:34:09.308012 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:34:09.309170 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:09.317090 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:09.325124 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:34:09.325917 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:09.327065 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:09.511736 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:09.512253 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:09.515165 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:09.516644 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:09.523490 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:09.618643 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:34:09.666568 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:34:09.693900 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:09.745311 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:09.745530 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:09.750353 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:09.750691 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:09.774787 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:09.854094 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:09.902014 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240502 14:34:09.909574 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:34:09.932389 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:09.947299 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:09.953481 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:09.958282 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:09.964833 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:09.993572 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:34:10.069821 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240502 14:34:10.102139 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:34:10.178962 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:10.187512 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:10.188557 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:10.192523 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:10.193095 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:10.251721 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:10.263830 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240502 14:34:10.276710 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:34:10.291002 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:34:10.318584 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:10.389051 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:10.390827 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:10.396432 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:10.400111 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:10.402483 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:10.539710 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:34:10.591136 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:10.598059 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:34:10.623708 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:34:10.635993 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:10.659436 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:10.659845 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:10.728113 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:34:10.762727 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:10.789896 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:10.830596 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:10.850045 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:10.882071 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:10.884159 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:10.885639 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:10.906704 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:34:10.982236 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
I20240502 14:34:11.051256 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:11.076543 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:11.092260 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:11.092878 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:11.093683 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:11.102627 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:34:11.199018 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 14:34:11.230643 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:11.300370 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:11.301143 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:11.303020 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:11.317055 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:11.324888 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:11.359028 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:34:11.370085 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:11.531173 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:11.542585 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:11.553761 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:11.557806 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:11.558669 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:11.628922 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:11.641479 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:34:11.687799 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:11.692808 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
I20240502 14:34:11.750324 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:11.767184 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:11.768294 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:11.769475 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:11.775882 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:11.915630 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:34:11.953012 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:34:11.953578 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:34:11.982882 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:11.983784 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:12.001487 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:12.003885 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:12.018430 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:12.043360 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:12.052593 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:34:12.099789 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:34:12.190609 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:12.203734 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:12.213402 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:12.219794 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:12.228705 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:34:12.232879 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:12.241500 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:12.266113 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:34:12.272846 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
I20240502 14:34:12.410665 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:12.416036 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:12.436411 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:12.441329 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:12.452950 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:12.555972 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240502 14:34:12.595324 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:34:12.600114 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:12.623659 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:12.633550 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:34:12.686748 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:12.694027 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:12.710201 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:12.712486 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:12.731449 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:12.806232 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:34:12.913508 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:12.921330 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:12.925453 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:12.929983 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
I20240502 14:34:12.932657 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:12.937748 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:13.017927 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:34:13.081976 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:34:13.132519 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:13.137465 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:34:13.144711 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:34:13.146977 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:13.166613 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:13.187775 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:13.191681 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:34:13.192746 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:13.216953 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:13.231209 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:34:13.363385 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:13.366004 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:13.370980 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:13.375393 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:13.389925 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:13.397652 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:34:13.437764 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
I20240502 14:34:13.582113 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:13.586313 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:13.608055 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:13.647234 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:13.658722 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:13.666114 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:13.720845 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:34:13.751968 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:13.901938 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:13.909152 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:13.923822 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:13.924777 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:13.945530 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:14.100963 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 14:34:14.121330 18768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240502 14:34:14.137866 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
I20240502 14:34:14.138793 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:14.147377 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:14.150380 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:14.150436 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:14.151505 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:14.193984 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:14.196347 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:34:14.202862 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:14.233037 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:34:14.306034 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:34:14.336057 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:34:14.348098 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:14.357030 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:14.368240 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:14.369498 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:14.375604 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:34:14.386286 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:14.463665 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:34:14.546672 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:14.574149 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:14.575116 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:14.590942 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:14.595490 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:14.690889 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:34:14.702760 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:34:14.713111 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:14.743919 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:14.793458 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
I20240502 14:34:14.816768 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:14.821730 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:14.831768 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:14.836585 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:14.844142 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:14.956486 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
I20240502 14:34:15.039090 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:15.039878 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:34:15.051932 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:15.063839 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:15.069483 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:15.070904 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:15.141959 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:34:15.215047 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:34:15.247429 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:15.256608 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:15.257284 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:15.262710 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:15.270018 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:15.283372 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:15.334751 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:15.433115 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240502 14:34:15.474179 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:34:15.478361 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:34:15.506932 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:15.507539 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:15.514453 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:15.534734 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:15.535569 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:15.535974 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:34:15.727720 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:15.747373 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:34:15.748520 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:15.755667 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:15.757367 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:15.764008 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:15.780432 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:15.842816 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:15.891459 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:34:15.904898 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:34:15.963620 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:15.970312 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:15.990541 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:15.995631 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:15.996459 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:16.070981 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240502 14:34:16.127434 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:34:16.207916 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:16.209116 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:16.229974 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:16.238514 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:16.239288 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:16.286484 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:16.286561 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:16.317962 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 14:34:16.419121 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:16.421080 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:16.430632 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:16.442981 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:16.447687 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:16.516767 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240502 14:34:16.584889 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:34:16.652227 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:16.652411 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:16.686666 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:16.689453 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:34:16.706424 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:16.718679 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:16.743744 18637 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.69:41119, user_credentials={real_user=slave}} blocked reactor thread for 49173.4us
W20240502 14:34:16.746989 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:34:16.865413 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:34:16.888473 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:16.888967 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:16.894838 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:16.907181 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:16.913933 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:16.915571 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:16.977515 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:34:16.995790 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:34:17.065199 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:34:17.099915 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:34:17.166522 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:17.171741 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:17.182653 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:17.184043 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:17.186009 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:34:17.207839 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:17.243782 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:17.290532 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240502 14:34:17.427608 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:17.466922 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:17.468809 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:17.469352 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:17.469815 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:17.473166 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:17.560479 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:17.563527 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:34:17.623615 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:34:17.702737 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
I20240502 14:34:17.745707 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:17.745729 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:17.746269 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:17.747176 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:17.747373 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:17.761518 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:34:17.945891 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:17.946030 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
I20240502 14:34:17.952199 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:34:17.953095 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:34:17.961444 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:17.967211 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:17.978799 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:34:17.985671 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:17.993739 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:18.005709 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:18.061790 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:34:18.097007 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:34:18.162649 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:34:18.196427 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:18.219345 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:18.225286 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:18.231981 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:18.246765 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:18.368613 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:34:18.432113 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:18.439399 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:18.440483 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:18.444213 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:18.450315 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:18.451268 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:18.491555 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:18.536515 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:34:18.671638 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
I20240502 14:34:18.691375 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:18.737277 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:18.737818 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:18.744205 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:18.748118 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:18.785207 27542 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:34:18.785431 27542 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:34:18.786315 27542 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
W20240502 14:34:18.789260 18636 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:18.800922 18637 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:34:18.801156 18637 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: 7ab6d395b30d4afb9821b61c9454430b, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:34:18.801471 27542 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:34:18.938177 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:18.954208 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:18.958120 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:18.970518 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:18.973219 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:19.002827 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:19.012085 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:19.169926 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:34:19.171590 18768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240502 14:34:19.173254 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:34:19.191062 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:19.192471 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:19.205585 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:19.217821 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 14:34:19.222018 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:19.246795 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:34:19.265478 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240502 14:34:19.281831 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:34:19.344388 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:34:19.395360 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:19.399565 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:19.411231 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:19.412407 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:19.418603 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:19.455533 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240502 14:34:19.465452 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:34:19.546978 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:19.554965 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:34:19.597232 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:34:19.606304 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:19.633606 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:19.641844 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:19.653575 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:19.666023 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:19.693579 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:19.740480 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:34:19.863413 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:19.872470 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:19.877427 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:19.889508 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:19.889509 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:19.996331 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:34:20.017902 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:34:20.112125 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:20.117307 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:20.122691 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:20.133929 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:20.137611 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:20.143155 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:20.191401 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:20.217365 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:34:20.306861 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:34:20.332327 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:20.342335 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:20.347141 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:20.354539 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:34:20.357735 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:34:20.362294 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:20.383900 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:20.388476 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
I20240502 14:34:20.544091 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:20.553076 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:20.555934 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:20.560106 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:20.577250 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:20.631072 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240502 14:34:20.633384 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:20.659685 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:20.737910 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:34:20.786569 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:20.797436 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:20.797689 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:20.798013 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:20.801474 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:20.866681 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:34:20.884371 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:34:21.007221 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:21.011979 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:21.014555 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:21.028021 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:21.042433 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:34:21.050865 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:21.053362 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:21.053947 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240502 14:34:21.228566 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:21.248256 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:21.255054 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:21.261727 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:21.267580 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:21.274538 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:21.399111 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 64 similar messages]
I20240502 14:34:21.459692 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:21.464696 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:21.483747 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:21.490339 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:21.522428 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:21.541504 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:21.692868 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:34:21.728458 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
I20240502 14:34:21.743649 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:21.750365 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:21.758909 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:21.769558 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:21.770881 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:21.815450 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:21.826607 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:34:21.841249 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:34:21.842651 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:34:21.875320 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 14:34:21.876199 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
I20240502 14:34:21.982067 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:21.989159 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:21.996958 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:22.014520 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:22.024956 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:22.050844 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:34:22.065147 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:34:22.066022 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:22.197291 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:22.209071 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:22.219090 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:22.224674 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:22.227829 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:22.330313 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:34:22.362937 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:34:22.421962 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:22.424417 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:22.436818 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:22.446208 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:22.449924 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:22.450521 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:22.598354 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:22.602137 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:34:22.645802 28281 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:34:22.645988 28281 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:34:22.646662 28281 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
W20240502 14:34:22.649458 18637 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:34:22.650104 18636 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:22.650243 18636 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: 7ab6d395b30d4afb9821b61c9454430b, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:34:22.650743 28281 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:34:22.657493 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:22.663494 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:34:22.678575 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:34:22.690053 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:22.698815 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:22.730592 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:22.765763 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:34:22.836992 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240502 14:34:22.868363 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:34:22.928251 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:22.929316 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:22.934474 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:22.946200 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:22.957703 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:22.960198 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240502 14:34:22.972790 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:23.029803 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:23.164429 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:23.164567 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:34:23.172686 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:23.175686 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:23.176538 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:23.189842 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:23.200989 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:34:23.229555 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:34:23.389335 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:23.391592 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:23.396768 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:23.400110 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:23.402941 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:23.474926 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:23.533967 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:34:23.579192 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:23.623821 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:34:23.633090 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240502 14:34:23.635088 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:23.639690 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:23.701695 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:23.723292 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:34:23.725250 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:23.847697 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:34:23.871201 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:23.911612 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:23.918764 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:23.921854 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:23.929781 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:23.931308 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:24.083333 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:24.129508 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:24.137703 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:24.146767 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:24.149892 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
I20240502 14:34:24.150712 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:24.161558 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:24.166944 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:34:24.185758 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240502 14:34:24.201294 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240502 14:34:24.213459 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:34:24.310655 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:34:24.347870 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:24.361045 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:24.362416 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:24.379611 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:24.380858 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:24.418462 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:24.477789 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:34:24.555279 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:24.565469 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:24.565873 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:24.566955 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:24.568359 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:34:24.591568 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:34:24.600374 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:34:24.602651 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:34:24.658879 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:24.805112 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:24.814989 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:24.823117 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:24.825229 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:24.827528 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:24.831465 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:24.859051 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:34:24.873554 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:34:25.028230 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:34:25.047798 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:25.053444 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:25.068550 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:25.073652 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:25.087071 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:25.106617 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:34:25.178879 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240502 14:34:25.189392 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:34:25.202678 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:25.257504 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:34:25.282618 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:25.284013 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:25.291110 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:25.294697 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:25.301645 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:25.362169 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:25.505822 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:25.521765 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:25.530750 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:25.533403 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:25.549677 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:25.582172 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:25.585579 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:34:25.596007 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:34:25.684749 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:34:25.772722 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
I20240502 14:34:25.788697 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:25.789767 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:25.814412 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:25.826627 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:25.900710 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:25.948179 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:25.994853 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240502 14:34:25.995628 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:26.010213 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:34:26.049629 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:26.051741 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:26.065531 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:26.072614 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:26.073635 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:26.170609 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:34:26.275985 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:26.281332 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:26.286645 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:26.291081 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:26.292480 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:26.372535 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:26.450873 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:34:26.482453 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:34:26.485679 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:26.489539 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:26.499441 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 63 similar messages]
I20240502 14:34:26.499614 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:26.505092 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:26.517392 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:26.557324 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240502 14:34:26.639083 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:34:26.667937 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
I20240502 14:34:26.734202 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:26.738921 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:26.757953 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:26.758318 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:26.769765 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:26.801071 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:34:26.807155 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:34:26.832351 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:26.947880 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 14:34:26.953406 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:26.954362 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:26.968816 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:26.972020 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:26.990363 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:27.003094 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:34:27.021965 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:27.162115 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:34:27.168099 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:27.169659 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:27.188680 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:27.198966 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:27.209311 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:27.288408 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:34:27.390189 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:27.391319 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:27.405310 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:27.409921 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:27.412125 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:27.418612 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:27.473351 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:34:27.550630 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:27.587859 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:34:27.608497 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:27.629642 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:27.629854 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
I20240502 14:34:27.636049 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:27.639472 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:27.650987 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:27.724715 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:34:27.830241 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:34:27.863606 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:34:27.870944 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:27.873466 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:27.880718 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:27.881841 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:27.882992 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:27.942657 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:27.969273 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:34:28.002777 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240502 14:34:28.054682 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:34:28.084820 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:28.113133 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:28.114290 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:28.122560 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:28.125011 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:28.133751 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:28.224666 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:34:28.299644 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
I20240502 14:34:28.344874 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:34:28.351832 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:34:28.352577 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:28.361454 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:28.368693 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:28.369644 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:28.453301 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:28.523885 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:28.555533 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:28.556316 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:28.560827 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:28.569445 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:28.583027 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:28.772364 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:28.784680 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:28.788121 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:28.788877 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:28.797437 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:28.807147 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:34:28.838944 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:34:28.947590 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:28.981513 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:28.996129 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:29.002313 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:29.004519 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:29.011714 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:29.014506 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:29.099081 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240502 14:34:29.124684 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:34:29.210191 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:29.213650 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:34:29.232511 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 98 similar messages]
I20240502 14:34:29.240028 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:29.257730 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:29.276295 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:29.298974 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240502 14:34:29.377800 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:34:29.424186 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:29.429880 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:34:29.433454 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:34:29.450299 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:34:29.457257 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:29.466661 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:29.467741 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:29.468595 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:29.473680 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:29.518317 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:29.617753 18549 maintenance_manager.cc:419] P 0283476f38234a96bfe72594576461be: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033353
I20240502 14:34:29.625595 18547 maintenance_manager.cc:643] P 0283476f38234a96bfe72594576461be: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.008s user 0.006s sys 0.001s Metrics: {"bytes_written":2023,"delete_count":0,"lbm_write_time_us":145,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":1113,"threads_started":1,"update_count":32,"wal-append.queue_time_us":1201}
I20240502 14:34:29.626307 18549 maintenance_manager.cc:419] P 0283476f38234a96bfe72594576461be: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.137320
I20240502 14:34:29.643772 18547 maintenance_manager.cc:643] P 0283476f38234a96bfe72594576461be: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.017s user 0.017s sys 0.000s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":14890,"cfile_init":3,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":2516,"dirs.run_cpu_time_us":282,"dirs.run_wall_time_us":1442,"lbm_read_time_us":377,"lbm_reads_lt_1ms":17,"lbm_write_time_us":340,"lbm_writes_lt_1ms":9,"mutex_wait_us":26,"reinsert_count":0,"thread_start_us":3054,"threads_started":2,"update_count":32}
W20240502 14:34:29.657740 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 14:34:29.664709 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:29.688047 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:29.719882 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:29.733975 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:29.752715 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:29.767149 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:34:29.825726 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:34:29.853379 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 14:34:29.918660 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:29.932585 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:29.934051 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:29.947897 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:29.954424 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:29.970729 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:29.985060 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:30.000252 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:34:30.182803 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:30.188989 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:30.195434 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:30.201783 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
I20240502 14:34:30.202095 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:30.216727 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:30.217362 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:34:30.235951 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:34:30.365794 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:34:30.386687 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:30.400851 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:30.413748 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:30.417307 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:30.429790 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:30.434962 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:30.524583 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:34:30.530506 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:30.596066 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:30.606763 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:30.632951 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:30.645346 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:30.665735 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:30.711845 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240502 14:34:30.779186 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:34:30.803315 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:34:30.836171 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
I20240502 14:34:30.846856 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:30.860690 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:30.870890 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:30.886049 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:30.887990 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:30.911094 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:30.914384 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:34:31.047469 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:31.079898 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:31.081812 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:31.098227 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:31.105993 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:31.118669 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:31.248842 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:34:31.248997 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:34:31.324154 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:31.342087 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:31.343235 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:31.344312 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:31.376745 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:31.428292 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:31.440433 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:31.524333 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:34:31.563258 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:31.575300 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:31.588783 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:31.592644 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:31.595674 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:31.803030 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 14:34:31.804126 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:31.808285 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:31.817082 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:31.835189 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:31.841463 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:31.855918 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:34:31.868357 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:31.916965 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240502 14:34:31.941063 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:31.979430 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:34:32.007514 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
I20240502 14:34:32.044903 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:32.046236 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:32.054368 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:32.068208 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:32.084312 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:32.153138 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:34:32.179880 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 14:34:32.261273 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:32.272693 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:34:32.276427 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:34:32.282474 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:32.283253 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:32.298938 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:32.324405 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:34:32.324546 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:34:32.413338 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:32.486164 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:32.493710 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:32.493754 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:32.498310 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:32.518678 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:32.522701 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:32.546391 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:34:32.715361 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:32.728312 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:32.734206 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:32.735149 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:32.742388 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:32.795941 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:34:32.852803 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:34:32.902926 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240502 14:34:32.939318 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:34:32.947117 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:32.957386 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:32.965577 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:32.978530 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:32.986737 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:33.018561 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:33.021420 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:33.216370 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:34:33.217768 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240502 14:34:33.234112 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:34:33.240592 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:33.254587 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:33.258774 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:33.263520 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
I20240502 14:34:33.265542 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:33.276297 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:33.338733 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:33.431376 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:34:33.470562 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:33.473709 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:33.484417 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:33.506727 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:33.542906 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:33.543272 18637 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.67:45409, user_credentials={real_user=slave}} blocked reactor thread for 63570.5us
I20240502 14:34:33.546530 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:33.662075 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:34:33.719370 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:34:33.721937 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:33.732403 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:33.741652 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:33.746222 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:33.748965 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:33.753468 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:33.788316 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:34:33.894073 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:33.939525 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:33.948760 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:33.967942 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:33.971778 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:33.984076 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:34.075217 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:34:34.165647 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:34:34.166810 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:34.181296 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:34.190335 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:34.197578 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:34.226166 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:34.291056 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 97 similar messages]
W20240502 14:34:34.292011 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:34.293110 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:34.341589 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
I20240502 14:34:34.401122 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:34.405596 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:34.429419 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:34.432657 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:34.434069 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:34.604771 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
I20240502 14:34:34.659931 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:34.671497 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:34.672019 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:34.673429 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:34.673152 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:34.702131 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 14:34:34.732265 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:34.825498 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:34:34.834888 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:34.838855 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:34:34.874679 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:34:34.877491 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:34.887436 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:34.898914 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:34.908780 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:34.925225 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:34.936292 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:34:34.937399 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:34:34.947016 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:34:35.104377 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:35.114388 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:35.115638 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:35.137791 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:35.139421 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:35.296550 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:35.306066 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:34:35.312512 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:35.336035 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:35.336560 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:35.338905 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:35.351455 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:35.382575 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:35.442123 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:34:35.470952 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:34:35.479578 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:34:35.513229 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240502 14:34:35.515314 18544 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 14:34:35.519482 18544 catalog_manager.cc:4573] TS a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 96662a3d9e1b4f03b05464ddd06b93d6: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:35.538790 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:35.570660 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:35.573307 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:35.581079 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:35.600783 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:35.681473 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:34:35.714753 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240502 14:34:35.749719 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:35.762287 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:35.809010 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:34:35.810194 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:35.819813 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:35.825939 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:35.842658 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:35.844622 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:35.911908 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240502 14:34:35.938584 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:34:36.036109 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:36.049482 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:36.051960 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:36.057567 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:36.067898 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:34:36.072520 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:36.103489 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:34:36.276515 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:36.281163 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:36.293792 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:36.293941 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:36.294929 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:36.300483 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:36.309903 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:36.473958 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:34:36.492313 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:36.498268 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:36.514732 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:36.519142 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:36.521402 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:36.636137 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:34:36.729511 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:36.736759 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:34:36.737866 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:36.749414 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:36.750610 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:36.751717 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:36.835143 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 14:34:36.873528 18544 catalog_manager.cc:4573] TS a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6a992d12aee7416096d1353533397386: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:36.883702 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:36.932706 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:34:36.939388 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:36.945103 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:36.946910 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:36.947798 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:36.953596 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:36.967902 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240502 14:34:37.104182 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
I20240502 14:34:37.154423 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:37.158949 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:37.159723 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:37.172258 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:37.178344 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:37.310299 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:37.357975 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:37.370482 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:37.374822 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:37.376878 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:37.376873 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:37.396739 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:37.452785 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:34:37.459674 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:34:37.520177 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:34:37.577762 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:34:37.602231 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:37.615554 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:34:37.618724 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:37.620635 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:37.620956 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:37.640367 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:37.708881 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:37.723233 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:34:37.834604 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:34:37.834769 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:34:37.869117 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240502 14:34:37.884030 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:34:37.897467 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:37.901217 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:37.903424 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:37.906579 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:37.909883 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:37.944151 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:34:37.951761 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:38.103538 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:38.107594 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:38.117063 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:38.137543 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:38.137872 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:38.146615 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:38.206964 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:34:38.245659 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
I20240502 14:34:38.338398 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:38.349464 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:38.359097 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:38.360059 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:38.366412 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:38.388499 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240502 14:34:38.389900 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:34:38.392699 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:34:38.407918 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:34:38.479066 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:38.586334 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:38.587292 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:38.597002 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:38.601661 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:38.613126 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:38.628504 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:34:38.630594 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:38.634846 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:34:38.823278 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:38.827809 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:38.841714 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:38.842550 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:38.853034 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:39.032135 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:39.034917 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:34:39.041159 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:39.060359 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:39.062624 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:39.077895 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:39.140069 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:39.182379 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:34:39.250352 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:39.254979 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:39.262192 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:39.271893 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:39.284833 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:39.375223 18544 catalog_manager.cc:4573] TS a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 0ffb6e582ef04fb38dd273c13febac58: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:39.381232 18768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240502 14:34:39.424111 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240502 14:34:39.435847 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:39.473802 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:34:39.503288 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
I20240502 14:34:39.504578 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:39.516351 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:39.529567 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:39.532882 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:39.553503 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:39.691776 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:39.729727 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:39.732437 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:39.737258 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:39.739849 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:39.760912 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:39.792680 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:34:39.883157 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:34:39.917508 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:34:39.958626 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:39.966961 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:39.975790 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:39.984588 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:39.993577 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:39.994839 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:34:40.024722 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:40.061803 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:34:40.121912 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240502 14:34:40.133803 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:34:40.157817 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:34:40.163625 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:40.180065 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:40.183406 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:40.192224 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:40.197100 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:40.213646 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:40.345070 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:34:40.374709 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:34:40.393617 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:34:40.407039 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:34:40.417272 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:34:40.425468 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:40.425683 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:40.429915 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:40.461843 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:40.611474 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:40.621042 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:40.642612 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:40.660197 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:40.682591 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:40.683456 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:40.687312 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:40.724772 30934 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:34:40.724972 30934 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:34:40.725683 30934 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
W20240502 14:34:40.726994 18636 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:40.746584 18637 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:34:40.746884 18637 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: 7ab6d395b30d4afb9821b61c9454430b, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:34:40.747221 30934 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:34:40.770479 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:34:40.779843 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240502 14:34:40.893785 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:34:40.903769 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:40.919806 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:34:40.924747 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:40.928143 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:40.932821 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:40.953083 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:40.990578 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240502 14:34:41.004634 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:34:41.070700 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:34:41.133713 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:41.137727 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:41.147941 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:41.152622 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:41.156944 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:41.179898 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:41.222796 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:41.380100 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:41.381649 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:41.388772 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:41.394063 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:41.394093 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:41.405644 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:34:41.451737 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:34:41.556646 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:41.638599 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:34:41.639389 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:41.657244 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:41.665431 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:41.704485 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:41.716033 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:41.765161 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240502 14:34:41.785534 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:34:41.860100 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240502 14:34:41.896824 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:41.902882 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:41.903390 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:41.904424 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:41.911543 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:42.043614 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:42.105131 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:42.112560 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:42.123804 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:42.139453 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:42.143976 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:42.158335 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:42.265390 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240502 14:34:42.295521 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:34:42.339004 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:42.354918 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:42.356627 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:42.358112 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:42.367795 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:42.472275 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:42.502714 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:42.559258 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:42.568631 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:34:42.569762 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:42.571067 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:34:42.573922 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 14:34:42.595425 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:42.612123 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:42.647194 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:34:42.728339 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240502 14:34:42.736594 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:34:42.800182 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:34:42.823590 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:42.827769 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:42.845651 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:42.849570 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:42.850620 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:34:42.865309 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:42.870334 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:42.938740 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:34:43.051967 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:34:43.062121 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:43.062577 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:43.070634 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:43.082890 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:43.084853 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:43.105204 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:43.119570 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:34:43.144990 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240502 14:34:43.161231 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:34:43.259336 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:43.265304 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:43.281102 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:43.301693 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:43.305703 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:34:43.328317 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:43.329372 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:43.457280 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:34:43.492820 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:43.493924 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:43.496400 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:43.499945 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:43.525705 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:43.586359 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:43.643818 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240502 14:34:43.664359 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:34:43.733156 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:43.741931 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:34:43.750397 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:43.751066 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:43.767557 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:43.768936 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:43.899518 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:34:43.924666 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:43.952278 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:43.953028 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:34:43.960202 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:34:43.981099 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:43.987313 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:43.987398 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:44.066591 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:44.220150 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:44.228675 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:44.232249 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:44.237079 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:44.237964 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:44.277370 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:34:44.406428 18768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 98 similar messages]
I20240502 14:34:44.429175 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:44.432296 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:44.439010 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:44.449908 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:44.454584 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:44.485411 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:44.493664 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:44.560956 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240502 14:34:44.655694 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
I20240502 14:34:44.669715 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:44.678004 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:44.683652 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:44.701782 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:44.715996 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:44.836405 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:34:44.918388 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:44.922848 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:44.924945 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:44.925882 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:44.933424 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:45.034513 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:34:45.035403 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:34:45.046684 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 14:34:45.081483 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:45.085913 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:45.133993 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:45.136893 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:45.151939 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:45.154567 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:45.154539 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:45.182793 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:34:45.228060 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240502 14:34:45.301863 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:34:45.358827 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:45.358873 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:45.359185 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:45.364936 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:45.385942 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:45.390636 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:34:45.409019 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240502 14:34:45.447099 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:34:45.472929 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:34:45.498804 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:45.540333 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:34:45.562317 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:45.579144 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:45.580374 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:34:45.594352 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:45.596767 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:45.631626 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:45.648030 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:45.660283 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:34:45.712747 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:34:45.861716 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:45.873994 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:45.879256 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:45.880440 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:45.880543 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:45.953299 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:34:45.994792 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:46.042616 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:34:46.063766 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:34:46.072049 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
I20240502 14:34:46.081751 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:46.083931 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:46.086318 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:46.094619 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:46.114019 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:34:46.175751 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:46.270799 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:34:46.282379 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:46.287345 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:46.306963 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:46.307386 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:46.315184 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:46.380007 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:34:46.479396 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:46.479811 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:46.501375 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:46.512624 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:46.539166 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:46.543563 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:46.565059 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:46.760248 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:46.760322 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:46.791810 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:46.796101 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:46.836098 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:46.894670 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:34:46.956727 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 14:34:46.992403 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:46.997397 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:47.002573 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:47.033344 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:47.047853 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:47.065030 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:47.082957 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:47.224303 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
I20240502 14:34:47.239528 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:47.245239 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:47.246048 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:47.251724 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:47.253602 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:47.436069 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:47.440565 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
I20240502 14:34:47.441890 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:47.467428 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:47.486838 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:47.490434 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:47.509083 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:34:47.529263 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:34:47.540468 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:47.579936 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:34:47.595484 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:47.634537 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:34:47.669528 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 14:34:47.694777 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:47.699612 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:47.740901 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:47.744383 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:34:47.751192 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:47.751245 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:47.761940 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
I20240502 14:34:47.949901 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:47.955262 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:47.956710 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:34:47.962069 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
I20240502 14:34:47.972342 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:47.975322 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:48.059801 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:34:48.098407 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:34:48.098919 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:48.116212 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:34:48.119606 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:48.157125 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:34:48.161648 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:48.168242 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:48.175493 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:48.186188 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:48.202783 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:48.215481 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:34:48.231184 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:34:48.234721 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:34:48.354661 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
I20240502 14:34:48.374104 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:48.387543 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:48.387650 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:48.388207 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:48.407963 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:34:48.440423 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:34:48.545830 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:48.569630 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:34:48.573166 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:48.589159 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:48.594828 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:48.637564 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:48.640434 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:48.666803 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:48.776084 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:34:48.856133 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:48.865609 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:48.876559 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:48.900297 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:48.914757 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:34:48.918884 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:49.044018 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:49.109088 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:49.109407 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:34:49.119149 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:34:49.126387 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:49.133399 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:49.144213 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:49.145056 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:49.350490 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:49.361903 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:49.366770 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:49.369081 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:49.374884 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:49.414105 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 97 similar messages]
W20240502 14:34:49.550246 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:49.561542 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:49.568140 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:49.569859 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:49.571911 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:49.581607 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:49.637341 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:49.800784 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:49.811017 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:49.817502 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:49.818090 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:49.826053 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:49.836942 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240502 14:34:49.887784 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:34:50.012672 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:50.028098 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:50.035363 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:50.037273 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:50.044822 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:50.057416 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:50.075340 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240502 14:34:50.078239 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:34:50.131451 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:34:50.133011 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:50.159577 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:34:50.167474 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:34:50.192543 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:34:50.243600 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:50.245503 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:50.256395 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:50.256816 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:50.259845 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:50.284404 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240502 14:34:50.441666 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:50.452046 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:50.456393 733 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:34:50.456789 733 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:34:50.458053 733 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
I20240502 14:34:50.459767 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:50.460947 18636 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:50.461407 18637 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:34:50.461524 18637 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: 7ab6d395b30d4afb9821b61c9454430b, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:34:50.462014 733 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:34:50.462085 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:50.466313 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:34:50.468430 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:34:50.469858 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:34:50.479332 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:50.513223 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:50.525315 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:34:50.579538 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:34:50.618346 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240502 14:34:50.628799 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:34:50.655762 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:50.661578 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:34:50.683108 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:34:50.689898 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:50.704349 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:50.704689 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:50.735636 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:34:50.743929 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:34:50.780252 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240502 14:34:50.898456 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:50.935345 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:50.936112 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:50.940690 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:50.942498 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:50.951604 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:51.032878 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:51.051060 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:34:51.113109 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:34:51.146229 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:51.148061 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:51.166754 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:51.170475 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:51.187074 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:51.195075 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:34:51.394048 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:51.400321 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:51.410935 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:51.413380 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:51.422976 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:51.515480 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:51.543422 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:51.570251 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:34:51.586181 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:34:51.599820 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:51.612347 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:51.640197 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:51.640846 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:51.657349 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:51.849052 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:51.858902 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:51.869375 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:51.874953 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:51.889508 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:51.969732 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:51.972963 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 14:34:52.064011 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:52.078462 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:52.081004 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:52.096026 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:52.104096 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:52.171010 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:52.323953 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:52.326851 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:52.335222 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:52.347142 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:52.353720 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:52.390156 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:34:52.512377 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:52.533247 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:52.548722 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:52.551216 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:52.553462 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:52.555019 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:52.555230 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240502 14:34:52.570151 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:34:52.666188 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:34:52.692546 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240502 14:34:52.714110 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:34:52.730613 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:34:52.759445 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:52.777774 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:52.784204 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:52.789454 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:52.790189 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:52.791174 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:52.962056 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:34:52.973009 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:34:52.993366 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
I20240502 14:34:53.015496 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:53.019189 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
I20240502 14:34:53.025125 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:53.043316 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:53.050235 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:53.064205 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:53.106868 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:53.142184 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:34:53.151253 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:34:53.151471 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:53.205533 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:34:53.252269 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:53.261706 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:53.267740 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:53.277141 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:53.291786 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:34:53.311082 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:53.395871 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240502 14:34:53.400274 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:34:53.429973 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:34:53.487982 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:53.493979 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:53.495703 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:53.506472 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:53.510118 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:53.563453 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:53.587162 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:34:53.702248 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:53.722570 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:34:53.722858 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:53.734838 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:53.739746 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:53.746091 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:53.759452 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:53.763015 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:34:53.924286 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:34:53.951697 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:53.955160 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:53.960304 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:53.963745 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:53.993817 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:54.031615 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:54.191988 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:54.192358 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:54.194780 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:54.205574 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:54.222018 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:34:54.222222 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:54.237751 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:54.378763 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:54.388808 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:54.389731 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:54.395283 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:54.403697 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:54.434760 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 100 similar messages]
I20240502 14:34:54.599174 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:54.603598 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:54.629923 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:54.656467 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:54.663375 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:54.675642 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:54.781168 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:34:54.795495 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:54.885499 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:54.888581 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:54.900918 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:54.916837 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:54.922014 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
I20240502 14:34:54.922008 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:55.054775 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:34:55.061442 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:55.111017 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:55.111472 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:55.125911 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:55.143141 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:55.150242 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:55.169204 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:34:55.218729 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:34:55.225471 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:34:55.226756 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:34:55.260985 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:34:55.272442 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240502 14:34:55.335589 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:55.356762 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:55.373795 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:55.389374 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:55.391000 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:55.429553 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
I20240502 14:34:55.441809 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:55.465533 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240502 14:34:55.508991 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:34:55.522497 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:55.628257 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:55.628768 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:34:55.640220 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:34:55.647274 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:34:55.648473 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:34:55.681982 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:55.682397 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:55.689397 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:55.760777 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:34:55.812348 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:34:55.813488 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:55.872450 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:34:55.914288 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:55.915022 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:55.922875 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:55.924189 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:55.924646 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:55.955966 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240502 14:34:55.959479 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:55.973371 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:34:56.146754 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:56.151454 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:56.153936 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:56.158505 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:56.173763 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:56.236922 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:56.280212 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:34:56.342757 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:34:56.362005 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:56.368636 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:56.373092 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:56.378648 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:56.383106 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:56.390266 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:34:56.391153 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:34:56.466899 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:34:56.570587 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:56.571381 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:56.587255 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:56.592990 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:56.616346 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:56.690279 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:56.804136 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:56.816960 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:56.827813 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:56.831004 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:56.832301 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:56.925443 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:56.978998 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 14:34:57.028942 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:57.032188 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:57.042802 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:57.043762 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:57.069244 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:57.203327 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:57.234069 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240502 14:34:57.243376 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:34:57.277678 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:57.279647 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:57.302887 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:57.316992 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:57.322326 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:57.376119 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:57.419153 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:34:57.487018 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:57.491192 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:57.503654 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:57.507046 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:57.511698 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:57.608940 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:34:57.656889 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:34:57.706969 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240502 14:34:57.719904 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 14:34:57.752701 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:57.754437 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:57.761716 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:57.773036 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:57.773557 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:57.778026 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:57.793864 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
I20240502 14:34:57.795328 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:57.807986 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240502 14:34:57.851233 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:34:57.990958 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:57.995887 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:58.012400 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:58.030905 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:34:58.035140 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:58.036237 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:58.073910 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:34:58.091939 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:34:58.153478 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:34:58.158500 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:34:58.237066 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:58.237730 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:58.241051 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:58.250156 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:58.257539 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:58.298161 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:34:58.315546 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:34:58.359755 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:58.368130 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:58.429421 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:58.455093 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:58.455410 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:58.457793 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:58.459156 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:58.480851 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240502 14:34:58.562608 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:34:58.640193 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:58.658496 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:58.678889 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:58.689749 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:34:58.704784 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:34:58.706491 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:58.774458 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:34:58.808398 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:58.861418 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:34:58.883297 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:34:58.884284 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:58.896152 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:58.902611 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:58.917867 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:58.929282 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:58.929298 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:59.121796 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:59.123801 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:59.129863 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:59.144573 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:59.150341 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:59.306564 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:34:59.332724 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:59.347047 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:59.349555 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:59.356139 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:59.358033 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:34:59.407104 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:34:59.481312 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 100 similar messages]
I20240502 14:34:59.555867 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:59.555891 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:59.572968 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:59.577780 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:59.602178 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:59.658488 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240502 14:34:59.667446 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:34:59.784883 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:34:59.785877 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:34:59.799798 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:34:59.803293 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:34:59.804348 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:34:59.919912 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:34:59.947429 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:00.015745 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:35:00.045466 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:00.047344 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:00.068735 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:00.069975 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:00.071137 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:00.130645 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:35:00.133008 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:35:00.227967 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240502 14:35:00.250460 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 14:35:00.279246 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:00.281545 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:00.284895 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:00.285625 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:00.297585 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:00.324704 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:00.330663 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240502 14:35:00.343914 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:35:00.401393 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 14:35:00.402287 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:35:00.450011 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:35:00.470227 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
I20240502 14:35:00.475373 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:00.480369 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:00.494381 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:00.496728 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:00.519187 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:00.556839 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:00.637562 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:35:00.704241 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:00.710319 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:00.715443 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:35:00.733927 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:00.735494 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:00.743597 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:00.764310 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:35:00.869900 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:00.919479 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:35:00.937095 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:00.938988 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:00.950770 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:00.953735 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:00.966964 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:00.970332 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:00.979164 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:35:01.070549 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
I20240502 14:35:01.158787 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:01.166947 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:01.181756 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:01.182509 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:01.197646 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:01.216226 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:35:01.313407 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:35:01.318043 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:35:01.342602 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:35:01.372370 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:01.382750 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:01.395851 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:01.398674 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:01.406240 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:01.417793 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:01.420364 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:01.500201 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:35:01.587247 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:01.617540 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:01.617916 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:01.631615 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:01.643234 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:01.756108 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:01.807719 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:35:01.850206 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:01.901655 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:01.902158 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:01.913221 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:01.919090 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:01.920931 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:01.979540 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240502 14:35:02.127034 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:02.133921 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:02.135859 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:02.145522 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:02.190214 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:02.295688 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
I20240502 14:35:02.341637 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:02.344338 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:02.355973 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:02.357836 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:02.368867 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:02.374833 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:02.415720 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:02.511046 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:35:02.566054 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:02.582736 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:35:02.583303 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:02.584381 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:02.590569 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:02.623289 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:02.677582 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:35:02.768343 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:35:02.794649 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:02.794956 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 27 similar messages]
I20240502 14:35:02.819263 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:02.828197 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:02.829046 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
I20240502 14:35:02.851323 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:02.853118 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:02.854411 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
I20240502 14:35:02.864779 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:02.893750 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:35:03.025624 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:03.072115 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:35:03.073961 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:35:03.085135 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:03.085342 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:03.103991 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
I20240502 14:35:03.104861 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:03.118165 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:03.118593 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:03.185833 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:03.216324 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:35:03.263818 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:35:03.304409 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:03.310740 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:03.311898 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:03.315143 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:03.317575 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:03.450852 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240502 14:35:03.511090 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:03.543679 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:03.550479 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:03.550657 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:03.556474 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:03.561408 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:03.575187 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:35:03.636178 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:03.676843 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:35:03.736860 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:35:03.784404 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:03.784531 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:35:03.800381 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:35:03.807121 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:03.812971 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:03.816751 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:03.837888 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:03.913892 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:35:04.031841 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:04.037093 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:04.045761 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:04.050920 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:04.051306 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:04.121855 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:04.137734 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:04.149235 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:35:04.226553 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:35:04.241468 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:04.252719 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:04.256707 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:04.265041 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:04.268440 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:04.465533 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:04.470880 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:04.478297 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:04.489231 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:04.499562 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:04.543705 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240502 14:35:04.573007 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:04.669800 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:04.677968 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:04.681828 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:04.696388 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:04.706740 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:04.706943 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:35:04.722884 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:04.896931 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:04.900969 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:04.916714 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:04.926532 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:04.926626 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:05.016275 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240502 14:35:05.081238 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:35:05.090827 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:05.124359 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:05.135711 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:05.137316 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:05.143349 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:05.150460 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:05.156440 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240502 14:35:05.166267 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:05.277802 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:35:05.287709 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
I20240502 14:35:05.342582 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:05.354873 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:05.365622 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:05.365828 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:35:05.381379 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:05.382048 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:05.468163 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:35:05.508001 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:35:05.551604 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:05.555238 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:05.558593 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:05.565897 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:05.569061 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:05.637341 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:05.637754 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:35:05.638636 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240502 14:35:05.662817 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:35:05.675733 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:35:05.695587 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:35:05.698700 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:05.784399 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
I20240502 14:35:05.822909 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:05.838245 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:05.844509 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:05.855579 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:05.859148 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:05.958650 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:35:05.998984 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:35:06.001708 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:06.047595 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:35:06.048254 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:06.051252 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:06.052296 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:35:06.070439 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:06.075137 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:06.080418 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:06.108773 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:35:06.158046 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:06.190418 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:35:06.275444 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:06.283994 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:06.294006 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:06.306525 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:06.308040 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:06.478132 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:06.478952 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:06.479480 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:35:06.494405 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:06.498919 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:06.499518 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:06.580619 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:06.591758 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:06.643390 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:35:06.715189 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:06.719130 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:06.736588 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:35:06.738471 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:06.739447 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:06.747601 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:06.931337 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:06.936851 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:06.950893 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:06.957139 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:06.966619 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:07.004858 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:07.007798 3558 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:35:07.007977 3558 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:35:07.008638 3558 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
W20240502 14:35:07.009425 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 14:35:07.010318 18637 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:35:07.026340 18636 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:07.026577 18636 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: 7ab6d395b30d4afb9821b61c9454430b, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:35:07.026871 3558 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:35:07.038164 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:07.162336 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:07.166648 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:07.170765 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:07.173591 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:07.224742 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:07.283776 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:35:07.385037 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:07.386898 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:07.391184 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:07.396472 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:07.411154 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:07.498596 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:07.541965 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240502 14:35:07.594146 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:07.598915 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:07.603552 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:07.607614 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:35:07.621430 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:35:07.628207 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:07.632369 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:07.645893 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:07.692152 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:35:07.746309 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240502 14:35:07.797664 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:35:07.798560 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:35:07.829655 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:07.852911 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:07.853806 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:07.864131 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:07.888573 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:07.937598 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240502 14:35:07.946499 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240502 14:35:08.033164 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:35:08.086633 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
I20240502 14:35:08.098011 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:08.099395 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:08.101642 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:08.113137 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:08.117271 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:08.129598 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:08.142623 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:08.203253 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:35:08.305389 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:08.306522 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:08.329697 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:08.333693 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:08.335633 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:35:08.339202 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:08.375763 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:35:08.411538 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:35:08.469681 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:35:08.498960 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:35:08.538551 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:08.556672 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:08.570341 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:08.588063 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:08.590123 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:08.597627 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:08.609954 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:08.630205 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:35:08.662968 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:35:08.731446 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:35:08.794211 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:08.816210 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:08.830989 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:08.831076 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:08.831547 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:08.915450 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:09.045888 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:09.064276 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:09.067031 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:09.075661 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:09.093817 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:09.094704 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:35:09.101207 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:09.277513 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:09.281448 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:09.289486 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:09.291241 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:09.305157 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:09.369951 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:09.418331 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:35:09.503659 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:35:09.515066 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:09.518560 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:09.525568 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:09.530762 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:09.534432 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:09.543972 18768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 103 similar messages]
W20240502 14:35:09.561832 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:09.759212 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:09.775012 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:09.779764 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:09.785334 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:09.794682 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:09.809847 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:09.906652 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
I20240502 14:35:09.980528 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:09.997568 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:10.000932 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:10.008728 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:10.012457 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:10.063269 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:35:10.113809 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240502 14:35:10.116219 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:10.223878 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:10.226083 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:10.231945 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:10.256635 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:10.258771 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:10.259322 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:10.262471 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:35:10.276017 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:35:10.363672 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:35:10.447263 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:10.452555 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:10.460250 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:10.460762 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:10.464933 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:10.516111 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:35:10.587128 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:35:10.653564 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240502 14:35:10.673784 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:35:10.675240 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:10.704887 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:10.707948 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:10.724727 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:10.725241 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:10.725744 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
I20240502 14:35:10.734606 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:10.748871 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240502 14:35:10.758601 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:35:10.759943 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:35:10.792280 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:10.798696 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:35:10.824644 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:35:10.914878 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:10.919167 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:10.928440 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:10.929461 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:10.938695 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:10.941612 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:35:10.972455 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:35:10.984683 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:35:11.122923 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:11.129596 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:11.143926 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:11.150708 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:11.153342 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:35:11.156100 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:11.163025 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:11.227726 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:35:11.380213 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:11.382778 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:11.388895 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:11.389703 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:11.403209 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:11.418169 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:11.498150 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:35:11.590155 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:11.591886 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:11.613415 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:11.615252 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:11.633702 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:11.654278 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:11.773447 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:11.829871 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:11.835444 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:11.866408 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:11.867935 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:11.900187 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:12.054793 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:12.060726 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:35:12.071280 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:12.073199 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:12.076016 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 14:35:12.083441 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:12.087863 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:12.127148 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:12.151042 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:35:12.228557 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:12.291435 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:12.294083 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:12.305037 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:12.310110 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:12.343780 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:12.442035 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240502 14:35:12.487056 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
I20240502 14:35:12.487408 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:12.488727 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:12.496209 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:12.501792 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:12.523860 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:12.552925 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:35:12.578928 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:35:12.650519 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:35:12.731606 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:12.734935 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:12.738677 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:12.766959 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:12.767032 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:12.771103 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:12.793545 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:35:12.803926 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:12.848577 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:35:12.975382 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
I20240502 14:35:12.990020 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:13.016999 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:13.026479 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:35:13.026613 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:13.035928 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:35:13.036044 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:13.037434 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:13.100916 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240502 14:35:13.197474 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:35:13.266791 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:13.267676 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:13.271646 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
I20240502 14:35:13.277892 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:13.286407 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:13.295899 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:13.315553 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:13.337030 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:13.396606 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:35:13.409250 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:35:13.411952 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:35:13.477028 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:13.490368 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:13.490893 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:13.496153 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:13.518649 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:13.699812 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:35:13.702462 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:35:13.705121 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:13.715667 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:13.716579 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:35:13.721009 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:13.731604 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:13.734678 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:13.744284 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:35:13.756175 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:13.801721 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:35:13.871625 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:13.956337 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:13.958474 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:13.970258 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:13.971935 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:13.982789 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:14.189380 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:14.191242 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:14.203792 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:14.206233 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:14.210148 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:14.237201 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:14.387972 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:14.394317 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:35:14.418694 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:14.431370 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:14.432924 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:14.437382 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:14.443437 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:14.450470 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:35:14.571782 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 100 similar messages]
I20240502 14:35:14.660372 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:14.664974 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:14.703765 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:14.708595 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:14.725082 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:14.751113 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:14.858044 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:14.933522 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
I20240502 14:35:14.936101 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:14.938695 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:14.945165 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:14.949111 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:14.951200 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:15.128795 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:35:15.137390 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:15.143292 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:15.146961 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:15.151888 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:15.159828 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:35:15.162638 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:15.181079 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240502 14:35:15.195883 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:15.331307 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:15.331830 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:35:15.332805 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:35:15.374334 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:15.381981 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:15.387145 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:15.387204 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:15.402215 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:15.525096 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:35:15.565820 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:35:15.571218 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:15.581112 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:15.585852 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:15.593029 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:15.609733 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:15.611984 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240502 14:35:15.637934 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:15.699628 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:35:15.724615 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240502 14:35:15.738709 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:35:15.762768 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:35:15.766644 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:15.798578 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:15.818955 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:15.819903 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:15.826805 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:15.853361 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:15.943588 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:35:15.974074 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:35:16.006080 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:35:16.018201 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:16.019953 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:16.023094 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:16.057022 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:16.061419 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:16.078487 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:16.079584 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240502 14:35:16.168522 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:35:16.180040 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:16.262205 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:16.265130 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:35:16.274230 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:16.296860 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:16.299396 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:16.320274 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:16.339217 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:35:16.403903 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:35:16.428083 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:35:16.523006 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:16.523866 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:16.534343 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:16.551692 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:16.554133 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:16.554848 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:16.747424 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:16.749574 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:16.758375 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:16.761536 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:16.766224 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:16.771633 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:16.959331 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:35:16.977365 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:16.982149 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:16.988212 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:16.998296 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:17.011750 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:17.019053 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:17.113005 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 14:35:17.131809 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:35:17.167687 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
I20240502 14:35:17.196914 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:17.206818 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:17.210574 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:17.226931 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:17.234097 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:17.312916 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:17.425469 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:17.426646 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:17.436672 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:17.441414 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:17.454600 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:17.545619 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:17.615845 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:35:17.648692 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:17.671948 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:17.672277 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:35:17.672652 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:17.683833 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:17.742671 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:17.873662 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:35:17.874559 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:35:17.890480 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:17.907727 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240502 14:35:17.929983 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:35:17.944914 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:17.947613 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:17.953953 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:17.957458 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:17.965206 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:17.966120 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:18.037482 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:35:18.082280 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240502 14:35:18.159165 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:35:18.166540 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:18.170159 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:18.183640 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:18.187301 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:18.189570 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:18.346329 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:18.366755 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:18.383085 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:18.391152 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:18.393065 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:18.404783 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:18.419826 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:18.425494 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:35:18.454792 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:35:18.468366 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:35:18.498270 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:35:18.522308 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240502 14:35:18.573062 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:35:18.671917 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:18.705911 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:18.707698 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:18.736302 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:18.747241 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:18.816550 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:35:18.832420 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:35:18.841041 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:35:18.881259 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:18.891270 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:18.922305 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
I20240502 14:35:18.954895 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:18.959534 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:18.966480 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:18.967217 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:18.975016 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:19.022053 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:35:19.187460 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:19.194001 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:19.198994 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:19.199467 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:19.212224 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:35:19.216087 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:19.313252 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:19.396812 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:35:19.404017 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:19.415059 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:19.416654 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:19.427917 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:19.436695 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:19.458169 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:19.586652 18768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 99 similar messages]
I20240502 14:35:19.656107 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:19.659333 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:19.665092 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:19.685930 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:19.707147 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:19.845156 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:19.854638 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:19.861966 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:35:19.863207 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
I20240502 14:35:19.909078 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:19.918216 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:19.925123 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:19.926225 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:19.933635 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:20.143555 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:20.149484 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:20.154114 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:20.155431 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:20.172693 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:20.254889 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:20.336007 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:35:20.350579 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
I20240502 14:35:20.365878 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:20.381701 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:20.393818 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:20.394162 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:20.397581 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:20.414772 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:20.448823 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:35:20.468907 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240502 14:35:20.487498 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:35:20.544374 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:35:20.598893 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
I20240502 14:35:20.599504 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:20.614064 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:20.615523 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:20.618686 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:20.643357 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:20.676800 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:35:20.724421 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:35:20.767971 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:20.772385 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:35:20.834903 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:35:20.835722 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:35:20.884714 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:20.888316 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:20.893487 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:20.898412 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:20.915479 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:21.029819 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:21.087405 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 14:35:21.100993 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:21.135047 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:21.138696 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:21.139699 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:21.142514 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:21.155058 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:35:21.190414 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:35:21.209432 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:35:21.264561 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240502 14:35:21.326583 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:21.360939 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:21.366237 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:21.385506 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:21.394403 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:21.398907 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:21.404431 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:35:21.446749 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:35:21.473116 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:35:21.519205 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:21.601435 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:21.610952 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:21.612990 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:21.623594 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:21.633757 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:21.716133 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:21.731866 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:35:21.844942 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:21.866757 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:21.875067 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:21.894757 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:35:21.896768 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:21.918576 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:22.096426 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:22.111178 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:22.113979 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:22.116814 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:22.120777 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:22.147163 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:22.208523 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 14:35:22.253410 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
I20240502 14:35:22.271881 6217 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:35:22.272086 6217 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:35:22.272807 6217 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
W20240502 14:35:22.274356 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:35:22.274495 18636 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:22.274792 18637 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:35:22.274935 18637 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: 7ab6d395b30d4afb9821b61c9454430b, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:35:22.275238 6217 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:35:22.319527 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:22.353569 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:22.357481 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:22.367977 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:22.367959 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:22.371034 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:22.567709 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:22.569803 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:22.574298 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:22.591600 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:22.594249 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:22.618327 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:22.677094 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:35:22.721161 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:35:22.745118 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:22.861439 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:22.861605 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
I20240502 14:35:22.879828 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:22.879999 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:22.899798 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:22.913427 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:22.945542 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:35:22.970981 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:35:22.972141 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:35:23.003536 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:23.126674 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:23.130730 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:23.131141 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:23.157972 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:23.163385 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:23.233486 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240502 14:35:23.235016 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:35:23.249522 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240502 14:35:23.266397 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:35:23.297091 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:23.359267 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:23.374076 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:23.380102 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:23.390719 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:23.393494 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:23.423671 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:35:23.487937 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:35:23.522138 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:35:23.537526 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:23.575858 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:23.576299 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:23.581790 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:23.602734 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:23.609573 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:23.694221 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:35:23.697700 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240502 14:35:23.716349 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:23.719480 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:35:23.836613 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:23.837386 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:23.840435 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:23.845259 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:23.855567 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:23.864593 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:35:23.926112 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:35:24.028250 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:35:24.038405 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:35:24.044615 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:24.050844 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:24.051817 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:24.057880 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:24.063391 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:24.131850 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:24.267045 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:24.283216 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:24.286109 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:24.296877 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:24.297412 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:24.340637 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:24.490404 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:24.499250 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:24.517539 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:24.518991 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:24.522555 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:24.539611 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:24.541810 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240502 14:35:24.612404 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240502 14:35:24.617043 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:35:24.747105 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:24.768137 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:24.769414 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:24.773164 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:24.781467 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:24.830996 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:35:24.936914 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:24.981081 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:24.983367 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:35:24.984627 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:24.989462 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:24.993736 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:25.010169 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:25.109023 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:25.213601 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:25.217388 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:25.244323 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:25.267120 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:25.269407 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:25.363992 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:35:25.377004 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:35:25.405481 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:25.443038 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:25.445828 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:25.449460 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:25.451457 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:25.468042 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:25.481813 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240502 14:35:25.551292 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:35:25.622499 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:25.695066 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:25.701156 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:25.733916 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:25.738821 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:25.741781 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:25.751228 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:35:25.772354 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:35:25.772491 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240502 14:35:25.878844 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240502 14:35:25.920749 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:35:25.942473 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:25.949249 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:25.953769 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:35:25.954681 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:25.956933 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:25.964594 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:25.992288 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:35:25.994423 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:26.010442 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:35:26.107441 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:26.140743 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:35:26.177031 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:26.180120 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:26.183467 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:26.183475 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:26.194108 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:26.234745 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:35:26.242038 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:35:26.349227 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:35:26.353713 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:35:26.397264 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:26.399106 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:26.406163 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:26.407714 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:26.409373 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:26.411439 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:26.593230 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:26.612864 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:26.614157 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:26.623881 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:26.632323 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
I20240502 14:35:26.643311 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:26.684191 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:26.742365 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:35:26.748649 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:35:26.838711 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:26.849066 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:26.851835 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:26.879994 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:26.900799 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:26.904260 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:27.080380 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:27.098124 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:27.099467 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:27.101926 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:27.103976 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:27.125732 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240502 14:35:27.186036 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:27.212695 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 14:35:27.270833 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:35:27.327342 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:27.330763 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:27.338938 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:27.348634 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:27.359549 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:27.394883 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:27.427485 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:35:27.430948 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:35:27.523803 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:27.524729 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:27.534476 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:27.544955 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:27.555867 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:27.770735 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:27.774283 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:27.779335 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:27.780927 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:27.784686 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:27.798194 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:27.919528 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:27.959162 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:35:27.968330 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:35:27.986660 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:27.999496 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:28.006026 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:28.006860 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:35:28.007952 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:28.022812 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:28.039947 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:35:28.095937 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:35:28.136430 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240502 14:35:28.162070 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240502 14:35:28.177455 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:35:28.220008 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:28.225126 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:28.258100 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:28.260811 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:28.276299 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:35:28.293201 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:28.340008 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:28.413168 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:35:28.415465 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:28.438733 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
I20240502 14:35:28.439589 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:28.449100 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:28.462198 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:28.482007 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:28.488139 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:28.495746 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:35:28.607447 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:35:28.665411 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:28.686576 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:28.701624 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:28.701628 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:28.703147 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:28.803979 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:35:28.863435 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:35:28.879175 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:35:28.895641 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:28.914628 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:28.947611 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:28.951283 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:28.954620 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:28.977473 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:28.978408 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:29.150410 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240502 14:35:29.167039 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:35:29.175102 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:29.181495 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:29.198889 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:29.198913 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:29.213559 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:29.293519 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:29.379779 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:29.406878 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:29.417788 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:29.418401 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:29.419587 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:29.427183 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:29.445611 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:35:29.612536 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 99 similar messages]
I20240502 14:35:29.613543 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:29.624887 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:29.655349 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:29.663297 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:29.675891 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:35:29.676587 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:29.692981 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:35:29.751446 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240502 14:35:29.802291 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:29.847030 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:29.850584 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:29.871685 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:29.885818 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:29.906700 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:29.911669 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:29.950995 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:35:30.103833 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:30.116003 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:30.121109 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:30.126032 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:30.126945 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:30.219903 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:35:30.265162 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:30.301167 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:30.314332 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:30.315276 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:30.331386 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:30.334524 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:30.343479 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:30.477385 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:35:30.498924 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:35:30.536914 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:30.550832 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:30.559142 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:30.561676 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:30.565465 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:30.594954 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:35:30.676999 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:30.687405 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:35:30.735018 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240502 14:35:30.749302 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240502 14:35:30.754958 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:35:30.823396 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:30.833125 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:30.845978 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:30.847313 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:30.881096 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:30.907788 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240502 14:35:30.943632 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:35:30.944401 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:35:30.962103 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:31.046620 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:35:31.067826 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:31.072903 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:31.074347 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:31.076825 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:31.096979 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:31.122185 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:31.211820 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 14:35:31.280645 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:31.280916 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:35:31.283037 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:35:31.291136 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:31.305398 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:31.319991 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:31.320816 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:31.416668 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:35:31.434608 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
I20240502 14:35:31.506552 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:31.506870 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:31.531078 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:31.531419 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:31.537206 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 14:35:31.548774 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:31.559281 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:31.625524 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:31.666863 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:35:31.748054 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:31.749053 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:31.760339 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:31.761267 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:31.769836 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:31.866227 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:35:31.954273 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:31.961596 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:31.967368 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:31.982405 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:31.992172 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:32.030018 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:32.101971 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:32.107224 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:35:32.200482 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:32.216575 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:32.234347 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:32.240340 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:32.243646 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:32.259213 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 14:35:32.307682 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240502 14:35:32.380285 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:35:32.404318 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:32.405328 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:32.420975 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:32.421458 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:32.433573 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:32.565424 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:32.578749 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:35:32.697293 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:32.717691 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:32.728844 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:32.730942 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:32.744211 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:32.763751 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:32.790899 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:35:32.913898 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:35:32.959489 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:32.967051 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:32.978309 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:32.992849 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:32.997666 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:33.003618 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:35:33.134780 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:35:33.156237 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:33.161351 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:33.181281 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:35:33.190279 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:33.205891 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:33.211602 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:35:33.215098 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:33.233917 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:33.246212 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
I20240502 14:35:33.256358 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:33.311725 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:35:33.328061 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:35:33.361956 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
I20240502 14:35:33.421917 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:33.434880 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:33.445770 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:33.447305 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:33.453970 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:33.533109 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240502 14:35:33.546288 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:35:33.660631 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:33.663829 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:33.694664 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:33.694837 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:33.698905 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:33.750548 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:33.764068 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:33.779794 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:35:33.869665 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:35:33.904971 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:33.913508 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:33.923085 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:33.931228 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:33.943936 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:35:33.948563 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:33.953872 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240502 14:35:33.960496 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:35:34.031338 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:35:34.138357 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:34.143656 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:34.151479 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:34.153954 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:34.167759 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:34.171700 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:34.205255 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:34.213723 7706 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:35:34.213923 7706 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:35:34.214599 7706 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
W20240502 14:35:34.217810 18636 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:34.218580 18637 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:35:34.218693 18637 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: 7ab6d395b30d4afb9821b61c9454430b, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:35:34.219204 7706 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:35:34.259769 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:35:34.372856 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:34.373407 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:34.388479 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:34.394115 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:34.400285 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:34.475569 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:35:34.580093 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:34.585453 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:34.593715 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:34.597419 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:34.613049 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:34.623116 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:34.677839 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 97 similar messages]
W20240502 14:35:34.682410 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:35:34.725653 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:34.845260 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:34.849093 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:34.858726 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:34.867796 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:34.868191 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:34.874354 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:35:34.923415 20561 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.72:35871, user_credentials={real_user=slave}} blocked reactor thread for 48265.8us
W20240502 14:35:34.925230 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240502 14:35:35.056114 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:35.080053 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:35.082056 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:35.086545 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:35:35.095790 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:35.109933 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:35.112282 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:35.232612 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:35:35.267684 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:35.321545 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:35.325027 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:35.335171 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:35.340631 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:35.341218 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:35.528431 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:35.541288 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:35.546252 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:35.552932 18544 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 14:35:35.560876 18544 catalog_manager.cc:4573] TS a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 96662a3d9e1b4f03b05464ddd06b93d6: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:35.570501 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:35.575062 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:35.660061 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:35.703269 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:35:35.704149 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240502 14:35:35.724505 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:35:35.737329 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:35.769505 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:35:35.780973 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:35.785141 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:35.802738 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:35.809098 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:35.820034 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:35.849267 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:35:35.853060 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240502 14:35:35.879211 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:35:35.895287 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:35:36.029842 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:36.030717 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:36.031304 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:36.034754 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:36.038534 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:36.056002 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240502 14:35:36.146840 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:35:36.169958 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:36.175705 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:36.198663 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:35:36.226753 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:36.239809 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:36.261952 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:36.291635 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:36.292800 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:36.299162 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240502 14:35:36.429036 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:35:36.432222 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:35:36.433456 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 14:35:36.464907 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:36.469512 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:36.489780 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:36.490490 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:36.494084 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:36.563679 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:36.646663 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:36.705466 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:35:36.706128 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:35:36.725301 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:35:36.726421 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:36.732537 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:36.762876 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:36.763960 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:36.766022 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:36.903102 18544 catalog_manager.cc:4573] TS a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6a992d12aee7416096d1353533397386: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:36.967156 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:36.969365 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:36.969472 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:36.971920 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:36.997370 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:37.010973 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:37.039330 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:35:37.161145 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:35:37.164611 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:37.173616 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:37.187698 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:37.190973 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:37.208381 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:37.259670 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 14:35:37.272527 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:37.379868 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240502 14:35:37.406334 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:35:37.418332 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:37.429795 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:35:37.429924 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:37.433086 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:37.433387 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:37.435842 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:37.439790 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:37.643321 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:37.671890 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:37.674404 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:37.675910 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:37.678812 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:37.687929 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:37.787189 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:35:37.873450 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:37.887900 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:37.894615 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:37.907887 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:37.923200 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:37.936072 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:38.012555 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:35:38.108516 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:38.113410 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:38.129546 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:38.137291 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:38.142158 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
I20240502 14:35:38.147468 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:38.172770 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:38.179806 8608 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:35:38.179991 8608 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:35:38.180639 8608 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
W20240502 14:35:38.182626 18636 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:38.189088 18637 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:35:38.189252 18637 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: 7ab6d395b30d4afb9821b61c9454430b, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:35:38.189517 8608 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:35:38.275315 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:35:38.324357 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:35:38.324995 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240502 14:35:38.342216 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:35:38.343466 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:38.357090 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:38.362525 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 14:35:38.363015 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:38.366716 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:35:38.370021 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:38.381932 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:38.455332 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:38.545923 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:38.550451 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:38.559276 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:38.574532 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:38.616047 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:38.616820 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:38.630118 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:35:38.728392 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240502 14:35:38.778376 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
I20240502 14:35:38.791201 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:38.799119 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:38.805701 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:38.821564 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:38.848245 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:38.873618 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:35:38.960767 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:35:38.960887 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:35:38.992120 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:39.003773 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:35:39.039294 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:39.055889 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:39.064286 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:39.065948 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:39.078219 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:39.110384 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:35:39.122932 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:39.264668 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:39.274494 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:39.280262 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:39.281461 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:39.299037 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:39.299412 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:35:39.381659 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:35:39.396605 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:39.406762 18544 catalog_manager.cc:4573] TS a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 0ffb6e582ef04fb38dd273c13febac58: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:39.503218 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:39.523838 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:39.527487 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:39.538743 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:39.541501 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:39.630715 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:39.676718 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:35:39.750864 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
I20240502 14:35:39.768357 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:39.768592 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:39.781942 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:39.795872 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:39.817768 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 100 similar messages]
I20240502 14:35:39.854048 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:39.870556 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:35:39.958323 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:39.985975 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:39.998926 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:40.003564 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:40.006582 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:40.013093 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:40.147881 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:35:40.173002 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:35:40.249701 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:40.253958 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:40.259806 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:40.267601 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:40.278553 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:40.291553 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:40.352756 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:40.382982 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:35:40.469061 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:40.478987 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:40.505257 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:40.506764 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:40.509529 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:40.648105 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240502 14:35:40.707018 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:35:40.732443 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:40.733565 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:40.755756 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:40.756258 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:40.757509 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:40.781452 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:35:40.850152 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:40.876734 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:35:40.935213 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:40.971694 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:40.973333 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:40.974639 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:40.986189 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:40.986702 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:41.032914 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240502 14:35:41.155272 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:35:41.186885 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:35:41.198557 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:41.198557 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:41.206851 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:41.222644 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:41.240059 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:35:41.242187 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
I20240502 14:35:41.262881 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:41.335232 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:41.399246 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:35:41.403188 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:35:41.414193 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:41.430301 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:41.434855 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:41.437594 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:41.439913 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:41.457589 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:35:41.461828 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 14:35:41.463243 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:41.470135 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240502 14:35:41.496834 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:35:41.670833 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:41.688446 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:41.720136 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:41.739794 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:41.764484 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:41.770200 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:41.775136 19032 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.69:41119, user_credentials={real_user=slave}} blocked reactor thread for 69576.8us
W20240502 14:35:41.779230 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:35:41.854908 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:35:41.948558 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:41.951992 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:41.955593 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:41.955906 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:41.972807 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:41.995821 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:42.074136 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240502 14:35:42.074786 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:35:42.177232 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:42.177235 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:42.181954 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:42.185778 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:42.211623 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:42.267704 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:35:42.324501 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 14:35:42.325428 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:35:42.334396 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:42.388476 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:35:42.423606 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:42.425056 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:42.442268 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:42.446311 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:42.450366 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:42.583750 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:42.634004 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:42.636554 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:42.675057 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:42.693135 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:42.712085 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:42.749795 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:35:42.750197 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:35:42.865818 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:42.881474 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:42.903003 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:42.904796 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:42.907496 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:42.923502 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:43.012614 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240502 14:35:43.039232 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:43.116930 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:43.124723 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:43.133020 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:43.140153 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:43.145705 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:43.181061 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:35:43.203258 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:35:43.334209 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:43.351451 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:43.352989 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:43.353423 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:43.357507 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:43.375276 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:43.504645 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:43.532977 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:35:43.573659 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:35:43.583951 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:43.592378 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:43.600287 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:43.609476 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:43.616906 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:43.646307 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240502 14:35:43.692497 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:35:43.696710 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240502 14:35:43.744537 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:35:43.817790 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:43.836596 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:43.837491 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:43.838318 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:43.849228 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:43.912561 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:35:43.917488 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:43.953075 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:43.953214 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:35:43.959290 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:35:44.030354 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:35:44.055699 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:35:44.070978 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:44.071831 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:35:44.081013 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:44.083156 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:44.096658 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:44.125496 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:44.129570 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
I20240502 14:35:44.325474 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:44.333590 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:44.340484 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:44.344449 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:44.345299 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:44.362502 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:44.416893 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:44.545598 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:44.555428 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:44.558882 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:35:44.564724 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:44.565769 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:44.579133 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:44.622776 9979 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:35:44.622941 9979 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:35:44.623597 9979 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
W20240502 14:35:44.626338 18636 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:44.626644 18637 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:35:44.626734 18637 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: 7ab6d395b30d4afb9821b61c9454430b, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:35:44.627179 9979 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:35:44.794811 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:44.796527 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:44.799635 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:44.804870 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:35:44.805593 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:44.823457 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:44.827193 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:44.845662 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240502 14:35:44.847270 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:44.885751 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240502 14:35:44.910974 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:35:44.977375 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:35:45.004415 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:45.008564 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:45.026818 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:45.047698 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:45.095731 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:45.254930 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:35:45.259333 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:45.270922 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:45.271303 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:45.272601 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:45.282783 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:45.288965 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:45.378744 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:35:45.396498 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:45.486004 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:45.495841 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:45.505712 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:45.507766 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:45.519634 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:45.547329 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240502 14:35:45.682078 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:35:45.738168 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:45.742743 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:45.744583 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:45.748021 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:45.786132 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:35:45.791342 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:45.829289 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:45.942219 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:45.943475 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:35:45.952579 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:45.958122 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:45.960743 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:45.966717 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:45.999624 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:46.059525 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
I20240502 14:35:46.158311 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:46.166751 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:46.167307 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:35:46.171617 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:46.185928 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:46.209856 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:46.245563 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:46.284466 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:35:46.324579 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:35:46.364847 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
I20240502 14:35:46.369697 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:46.387493 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:46.390377 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:46.395001 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:46.396054 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:46.407110 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:46.487568 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:35:46.487718 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:35:46.488479 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:35:46.499300 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:35:46.583371 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:35:46.604000 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:46.605896 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:46.612057 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:35:46.621928 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:46.624998 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:46.629029 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:46.792418 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240502 14:35:46.812012 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:46.813735 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 14:35:46.842060 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:46.842913 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:46.843477 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:46.849804 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:46.867617 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:46.970393 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:47.062741 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:47.067621 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:47.081470 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:47.084033 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:47.101258 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:47.244469 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:35:47.249555 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:47.282327 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:47.285353 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:47.299952 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:47.309594 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:47.316201 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:35:47.320295 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:47.342207 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240502 14:35:47.436091 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240502 14:35:47.437013 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:35:47.446372 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:47.469020 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:35:47.502874 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:47.516530 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:47.519047 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:47.525171 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:47.541384 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:47.745290 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:35:47.754351 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:47.763844 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:47.775535 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:47.788586 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:47.793599 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:47.826181 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:47.872519 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:47.991727 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:47.997454 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:48.002911 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:48.010596 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:48.017168 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:48.155668 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240502 14:35:48.183995 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:35:48.226279 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:48.227038 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:48.262964 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:48.263152 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:48.279551 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:35:48.294615 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:48.381865 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:48.391191 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:48.394091 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240502 14:35:48.433610 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:35:48.460057 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:48.475921 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:48.478734 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:48.496009 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:48.509035 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:48.524729 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:35:48.608736 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240502 14:35:48.610594 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:35:48.667554 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:48.683400 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:48.699436 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:35:48.703759 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 14:35:48.735440 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:48.754698 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:48.763150 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:48.763971 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:35:48.790838 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:48.790812 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:35:48.791563 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:48.950567 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:48.956946 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:48.959406 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:48.960845 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:48.974725 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:35:48.978806 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:49.064829 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:35:49.118113 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:35:49.162366 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:49.169414 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:49.174432 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:49.177443 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:49.188431 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:49.343047 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:49.355249 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:49.409109 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:35:49.416649 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:49.424230 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:49.426784 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:49.431964 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:49.462152 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:49.478044 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
I20240502 14:35:49.616744 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:49.626852 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:49.637785 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:49.651742 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:49.655267 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:49.657001 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:35:49.714251 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240502 14:35:49.734086 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:35:49.767596 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:35:49.850886 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:49.855371 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240502 14:35:49.875656 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:49.876463 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:49.883487 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:49.887449 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:49.894253 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:49.898380 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:49.957024 19031 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.5.107.72:35871, user_credentials={real_user=slave}} blocked reactor thread for 62811.6us
W20240502 14:35:50.040879 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:35:50.093837 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:50.102051 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:50.110894 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:50.112025 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:50.125469 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:50.288023 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:50.294207 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:50.294854 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:50.304303 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:50.310256 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:50.324204 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:50.355316 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:50.360826 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:35:50.523679 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
I20240502 14:35:50.542831 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:50.550141 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:50.550436 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:50.551983 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:50.556636 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:50.693025 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:35:50.702172 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:50.738333 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:35:50.758215 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:50.781150 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:35:50.800732 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:50.804251 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:50.807399 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:50.807698 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:50.838019 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:50.847478 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:35:50.869426 20562 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.67:45409, user_credentials={real_user=slave}} blocked reactor thread for 47679.6us
W20240502 14:35:50.933326 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240502 14:35:50.947152 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:35:51.009028 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
I20240502 14:35:51.043135 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:51.056593 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:51.061748 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:51.067953 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:51.070114 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:51.178812 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:35:51.179868 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:35:51.229116 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:51.273960 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:51.281347 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:51.281399 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:51.290007 10321 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:35:51.290171 10321 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:35:51.290920 10321 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
W20240502 14:35:51.294955 18636 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:51.295753 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:51.296521 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:51.304883 18637 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:35:51.304981 18637 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: 7ab6d395b30d4afb9821b61c9454430b, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:35:51.305534 10321 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:35:51.369140 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:51.400614 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:35:51.467482 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:51.474376 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:51.483498 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:51.495617 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:51.498277 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:51.536574 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:35:51.537511 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:35:51.609809 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:51.651537 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:35:51.712217 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:35:51.734385 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:51.738739 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:51.742810 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:51.747773 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:51.749446 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:51.830040 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:51.883152 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 14:35:51.930049 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
I20240502 14:35:51.958703 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:51.967536 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:51.981722 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:51.985404 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:51.988430 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:52.036727 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:52.072916 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:35:52.103833 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:35:52.201987 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:52.221001 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:35:52.221689 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:52.222084 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:52.222909 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:52.227233 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:52.317762 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:52.372229 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
I20240502 14:35:52.413267 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:52.417990 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:52.432782 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:52.443702 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:52.453719 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 14:35:52.456691 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:35:52.459304 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:52.555997 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:52.560032 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:35:52.618180 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:52.647607 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:52.664871 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:52.680414 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:52.685320 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:52.791571 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:52.872671 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:52.886380 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:52.895536 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:52.898922 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:52.902899 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:52.922384 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240502 14:35:53.038548 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:35:53.100486 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:53.106606 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:53.117769 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:53.119283 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:53.121474 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:53.127312 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:53.217325 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:53.246706 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240502 14:35:53.289000 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:35:53.326278 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:53.335937 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:53.336437 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:53.345433 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:53.350895 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:53.383565 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:35:53.409332 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:35:53.467000 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:35:53.476223 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240502 14:35:53.483014 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:35:53.555557 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:35:53.566653 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:53.572450 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:53.587503 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:53.603153 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:53.613245 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:53.638830 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:53.652318 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:35:53.682005 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:35:53.685626 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:53.817907 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:53.826272 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:53.834859 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:53.837815 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:35:53.848335 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:53.848824 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:53.852371 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:35:54.048957 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:54.068224 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:54.083736 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:54.084273 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:54.090842 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:54.108657 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:54.127871 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:35:54.269872 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:54.294409 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:54.301031 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:54.309865 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:54.330969 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:54.333528 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:54.377805 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:35:54.495939 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:54.508536 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:54.515123 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:54.522686 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:54.533419 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:54.550875 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240502 14:35:54.587337 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:35:54.611465 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:54.642051 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240502 14:35:54.655602 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:35:54.739459 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:54.752713 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:54.754891 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:54.765321 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:54.770961 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:54.798396 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:54.799665 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:35:54.867522 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 98 similar messages]
I20240502 14:35:54.944109 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:54.951093 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:54.953462 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:54.957875 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:54.973067 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:55.067353 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:55.148447 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:35:55.163717 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:55.164223 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:55.167212 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:55.171133 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:55.175696 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:55.296002 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:55.382778 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:55.392793 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:55.399173 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:55.410425 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:55.412074 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:55.417918 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:35:55.487396 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
I20240502 14:35:55.598610 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:55.638388 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:55.645260 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:55.666213 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:55.670120 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:55.686213 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:55.705929 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:35:55.729387 18769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.69:41119, user_credentials={real_user=slave}} blocked reactor thread for 44243.9us
W20240502 14:35:55.732084 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:35:55.779712 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:35:55.837330 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240502 14:35:55.862107 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:55.922955 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:55.930806 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:35:55.937510 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:55.939090 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:55.946520 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:55.950968 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:55.968715 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240502 14:35:55.978307 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:35:55.983805 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:35:56.052140 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:35:56.060067 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:35:56.063052 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:56.143185 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:56.146868 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:56.153604 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:56.159243 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:35:56.173683 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:56.174943 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:56.213573 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:35:56.348978 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:56.352135 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:56.361994 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:56.368294 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:56.369290 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:56.417429 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:56.560954 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:56.571835 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:56.575922 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:56.575996 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:56.610359 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:35:56.614488 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:56.631060 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:56.795878 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:35:56.796612 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:56.813275 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:35:56.814527 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:56.815541 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:56.823282 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:56.841015 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:57.008147 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:35:57.012445 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:57.038760 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:57.048889 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:57.059134 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:57.067409 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:35:57.067595 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:35:57.069722 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:57.074574 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:57.081637 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:57.261065 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
I20240502 14:35:57.261538 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:57.272953 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:57.275274 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:57.276150 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:57.278577 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:57.312141 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:35:57.374858 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240502 14:35:57.477802 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:35:57.499519 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:57.505151 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:57.506294 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:57.515302 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:57.521991 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:35:57.563197 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:35:57.592607 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 14:35:57.633986 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:57.737787 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:57.755216 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:57.755424 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:57.772130 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:57.775733 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:57.833907 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240502 14:35:57.906863 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:57.925382 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:35:57.966312 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:57.968600 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:57.984222 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:57.986773 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:57.994024 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:58.004141 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:35:58.021443 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:58.041222 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:35:58.147266 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
I20240502 14:35:58.212463 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:58.215945 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:58.218847 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:58.243350 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:58.243626 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:58.442418 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:58.443521 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:58.448218 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:58.449517 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:58.452108 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:58.455242 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:35:58.480000 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:58.488611 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:58.494920 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:35:58.524343 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:35:58.552245 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:35:58.557196 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:35:58.661985 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:58.666960 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
I20240502 14:35:58.673702 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:35:58.686296 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:35:58.699563 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:58.704461 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:35:58.710278 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:58.762605 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:58.905823 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:58.910176 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:58.923555 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:58.928309 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:35:58.928467 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:58.934062 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:58.960867 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:58.990689 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:59.015059 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 14:35:59.146237 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:59.147720 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:59.148200 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:59.148835 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:59.149438 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:59.213331 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:35:59.350395 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:35:59.362044 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:59.363361 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:59.371995 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:59.375344 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:59.378451 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:59.431283 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:35:59.478545 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20240502 14:35:59.491909 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:35:59.561043 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:35:59.571513 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:59.572072 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:59.593137 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:35:59.602115 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:59.606600 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:35:59.630511 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:35:59.747138 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:35:59.765560 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:35:59.827261 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:59.832437 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:35:59.841864 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:35:59.846015 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:35:59.853482 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:35:59.870563 18768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240502 14:35:59.989634 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
I20240502 14:36:00.058164 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:00.063768 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:00.071596 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:00.073578 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:00.078941 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:00.082191 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:00.184724 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:36:00.288457 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:00.296528 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:00.311844 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:00.313894 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:00.323503 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:00.349545 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:00.357041 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:36:00.487565 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:00.489429 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:00.501498 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:00.502871 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:00.517848 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:00.526837 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20240502 14:36:00.592778 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:36:00.640240 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:00.645541 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
I20240502 14:36:00.725828 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:00.745632 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:00.761891 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:00.763132 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:00.764319 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:00.764367 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:00.769654 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:36:00.914611 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:36:00.941570 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:36:00.959101 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:00.975845 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:00.982969 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:00.985937 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:00.986697 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:01.066547 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:01.161468 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:01.165084 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:36:01.175251 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:01.181561 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:36:01.185950 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:01.186879 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:01.191823 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:01.204963 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:01.205246 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:36:01.325446 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20240502 14:36:01.367884 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:36:01.408123 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:01.411592 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:01.438957 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:01.445365 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:01.465238 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:01.507032 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:36:01.515820 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:36:01.557946 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:01.617415 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 14:36:01.618539 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:01.682888 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:01.701750 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:01.724843 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:01.731069 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:01.737607 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:01.895735 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:36:01.924670 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:36:01.940582 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:01.943751 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:01.954828 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:01.955790 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:01.956306 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:01.987457 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:36:02.026319 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:36:02.028815 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:36:02.092557 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:02.126458 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20240502 14:36:02.153870 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:02.184087 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:02.191359 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:02.196413 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:02.202874 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:02.221552 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:02.297622 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:36:02.393481 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:02.404186 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:02.406419 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:02.423403 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:02.426086 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:02.468497 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
I20240502 14:36:02.612551 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:02.619625 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:02.637492 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:02.657757 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:02.666543 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 14:36:02.667567 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:36:02.695235 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:02.700824 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:02.760435 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:36:02.769359 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:02.872257 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:02.898299 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:02.901012 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
I20240502 14:36:02.905153 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:02.908257 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:02.933502 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:02.949252 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
W20240502 14:36:02.960036 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:36:03.094519 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:03.098174 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:03.101631 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:03.105445 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:03.110247 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:03.276679 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:03.325404 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:03.353634 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:03.359017 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:03.359937 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:03.365489 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:03.374714 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:03.431605 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:36:03.463002 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:36:03.529029 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:36:03.576016 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:03.577574 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:03.584605 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:03.595403 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:03.597349 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:03.602089 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:36:03.653926 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:03.694041 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:36:03.698710 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:36:03.800417 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:03.801890 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:03.809995 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:03.823329 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:03.827713 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:03.830161 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:03.882863 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:36:03.914260 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:36:03.988464 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:36:04.022780 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:04.031071 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:04.031880 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:04.058876 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
I20240502 14:36:04.066284 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:04.066565 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 14:36:04.072010 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:04.242076 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:04.276782 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:04.278259 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:04.285452 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:04.286072 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:04.286725 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:04.354889 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:36:04.371361 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:04.440765 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:36:04.486126 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:04.499362 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:04.500113 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:04.503741 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:04.513947 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:04.562116 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20240502 14:36:04.596177 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:36:04.677795 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:04.688869 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:36:04.712083 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:04.714062 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:04.775123 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:04.776152 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:04.779126 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:04.782367 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:04.852636 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:36:04.861629 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20240502 14:36:04.908958 18768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 99 similar messages]
I20240502 14:36:04.968006 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:04.973392 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:04.986450 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:04.986985 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:04.992388 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:05.140080 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:36:05.194132 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:05.202152 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:05.212179 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:05.213043 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:36:05.232411 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:05.245643 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:05.261163 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:05.278370 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
I20240502 14:36:05.281262 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:05.431442 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:05.454967 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:05.458775 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:05.465654 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:05.482862 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:36:05.487123 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:05.488662 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20240502 14:36:05.652927 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:05.661055 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:05.674338 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:05.680909 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:05.709570 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:05.725473 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:05.747611 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:36:05.787163 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:05.912674 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:05.920567 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:05.937824 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:05.938230 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:05.955694 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:06.009445 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:36:06.066386 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:36:06.123656 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:36:06.140890 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:36:06.166635 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:36:06.172096 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:06.179370 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:06.191350 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:06.196393 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:06.201195 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:06.226572 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:06.228845 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:06.349475 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:36:06.401512 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:06.403023 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:06.423480 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:06.423530 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:06.424131 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:06.486765 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:36:06.527923 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:36:06.591393 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:36:06.637693 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:06.642096 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:06.654436 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:06.655310 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:06.663559 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:06.663810 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20240502 14:36:06.690254 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:36:06.697242 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:36:06.765497 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:06.849239 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:06.877426 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:06.878062 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:06.882081 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:06.894133 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:06.901929 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:07.090715 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:07.104400 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:07.114789 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:07.129113 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:07.147516 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:07.165107 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:07.170770 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20240502 14:36:07.204952 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:36:07.277923 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:36:07.279364 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:36:07.311682 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:07.331828 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:07.332906 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:07.333238 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:07.334990 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:07.375025 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:36:07.375973 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:07.496073 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
I20240502 14:36:07.574105 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:07.582909 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:07.587236 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:07.587481 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:07.606189 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:07.611627 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:07.723975 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:36:07.771075 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20240502 14:36:07.772886 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:07.811138 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:07.814081 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:07.819144 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:07.823264 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:07.841146 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:07.850814 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 14:36:07.851668 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:36:08.000671 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20240502 14:36:08.003907 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:08.031425 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:36:08.052532 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:08.060226 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:08.072865 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:08.077651 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:08.078874 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:08.300518 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:08.300686 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:08.303337 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:08.305096 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:08.305367 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:08.326464 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:36:08.333947 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:08.411453 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:36:08.493094 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:08.503960 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:08.510098 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:08.518440 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:08.520486 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:08.602306 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:08.602450 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:36:08.612347 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:36:08.734946 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:08.739897 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:08.756585 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:08.768694 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:08.774202 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:08.776361 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:08.777263 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:36:08.821166 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:36:08.830883 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:36:08.938552 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:08.943588 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:36:08.946398 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:08.961520 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:08.962383 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:08.967592 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:09.028892 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:36:09.051615 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:36:09.185982 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:09.189818 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:09.199729 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:09.201354 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:09.213479 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 14:36:09.215288 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:09.218111 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:09.246469 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:36:09.318912 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:09.396018 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:09.401100 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:09.405112 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:09.413635 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:09.426520 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
I20240502 14:36:09.428717 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:09.436453 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:36:09.597954 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:09.636399 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:09.641589 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:09.670653 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:09.678494 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:09.726639 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:09.770864 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
I20240502 14:36:09.775957 14576 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:36:09.776124 14576 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:36:09.776744 14576 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
W20240502 14:36:09.782680 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:36:09.789613 18637 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:36:09.790119 18636 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:09.790221 18636 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: 7ab6d395b30d4afb9821b61c9454430b, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:36:09.790632 14576 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:36:09.827046 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:36:09.864027 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:09.924856 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240502 14:36:09.929143 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
I20240502 14:36:09.929873 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:09.930107 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:09.937721 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:09.947533 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:09.950285 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:36:09.990711 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:10.002498 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:10.150007 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:10.155018 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:10.162125 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:10.167100 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:10.182940 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:10.199043 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20240502 14:36:10.285262 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
I20240502 14:36:10.372023 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:10.379273 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:10.385486 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:10.394012 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:10.403793 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:10.411092 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:10.428633 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:36:10.433995 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:10.536152 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:36:10.579269 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:10.583524 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:10.606621 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:10.607069 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:10.609042 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:10.821255 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:10.825783 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:10.833906 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:10.847060 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:10.856122 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:10.949436 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:10.967823 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:10.996678 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:36:11.028595 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:11.043102 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:11.048938 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:11.066541 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:11.069634 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:11.175712 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:36:11.193852 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:36:11.196477 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:36:11.197474 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:36:11.198105 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:36:11.222512 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:36:11.266443 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:11.283275 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:11.283573 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:11.284518 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:11.306660 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:11.343379 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:36:11.366725 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:11.377254 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:11.490464 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:11.491464 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:11.512409 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:11.522347 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:11.527973 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:36:11.541469 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:11.562135 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:36:11.698205 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
I20240502 14:36:11.731464 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:11.731837 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:11.739629 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:36:11.740059 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:11.750385 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 14:36:11.762162 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:11.801903 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:11.813280 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:11.835278 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:36:11.847646 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:11.975657 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:11.993367 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:11.996291 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:12.010347 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:12.014066 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:12.108505 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:36:12.209491 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:12.227941 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:12.249931 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:12.252703 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20240502 14:36:12.253980 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:12.258373 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:12.271601 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:12.290323 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:36:12.331097 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:12.405678 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 14:36:12.434729 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:12.435920 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:12.457273 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:12.457417 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:12.459482 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:12.480247 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20240502 14:36:12.559387 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20240502 14:36:12.605702 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:36:12.649734 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:36:12.670446 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:12.693240 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:12.714668 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:12.719652 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:12.721251 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:12.762624 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:12.766566 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:12.857431 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
I20240502 14:36:12.938656 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:12.942004 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:12.942582 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:12.958262 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:12.969619 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 65 similar messages]
I20240502 14:36:12.972121 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:12.982208 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:36:13.151316 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:13.156867 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:13.167443 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:13.174705 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:13.179633 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:13.184528 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:13.199569 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:36:13.244696 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:13.300231 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:36:13.379442 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:13.381466 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:13.395845 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:13.399186 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:13.404629 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:13.461524 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:36:13.593672 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:13.595769 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:13.611416 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:13.615514 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:13.617856 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:13.680636 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:36:13.694921 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:36:13.698627 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:36:13.726648 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:36:13.757252 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:13.804006 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:13.833077 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:13.847010 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:13.851172 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:13.854372 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:13.867115 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:13.876397 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:36:13.943708 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:36:14.047689 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:36:14.072393 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:14.092295 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:14.099592 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:14.112572 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:14.117069 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:14.254175 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:36:14.256109 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 14:36:14.283535 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:14.286928 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:14.292105 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:14.301615 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:14.306706 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:14.308071 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:14.339525 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:36:14.363387 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:14.391345 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20240502 14:36:14.480547 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:36:14.499480 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:14.501436 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:14.507851 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:14.516568 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:14.528275 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:14.687912 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:14.710631 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
I20240502 14:36:14.758729 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:14.758795 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:14.759529 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:14.769474 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
I20240502 14:36:14.771967 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:14.772330 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:14.808586 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:36:14.859465 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:14.915248 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:36:14.933707 18768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 104 similar messages]
W20240502 14:36:14.950023 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
I20240502 14:36:14.963797 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:14.974403 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:14.987747 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:14.997797 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:15.026813 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:15.112969 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
I20240502 14:36:15.184796 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:15.190992 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:15.199051 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:15.216517 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:15.221279 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:15.238188 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:15.329063 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:15.381318 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:36:15.416473 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:15.431334 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:15.431809 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:15.433214 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:15.442409 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:15.541098 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:36:15.591495 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:15.626189 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:15.632938 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:15.660126 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:36:15.679973 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:15.684554 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:15.709223 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:15.821020 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:15.825650 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:36:15.912528 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:15.916442 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:15.928874 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:15.929765 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:15.938241 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:15.996454 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:16.002813 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:36:16.026507 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:36:16.157297 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:16.160141 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:16.160199 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:16.175415 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:16.194435 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:16.250886 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:36:16.267243 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:36:16.271533 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:36:16.378018 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:16.382091 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:16.397855 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:36:16.407636 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:16.408208 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:16.413367 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:16.425668 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:16.466231 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:16.494807 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:36:16.588475 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:16.604463 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:16.607962 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:16.609431 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:16.639015 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:16.642465 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:36:16.697041 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:36:16.762341 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:36:16.782385 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20240502 14:36:16.787492 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:36:16.865288 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:16.868631 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:16.871181 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:16.892987 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:16.896811 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:16.901818 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:36:16.902767 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:16.908428 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:17.090193 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:17.092458 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:17.107019 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:17.109845 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:17.111166 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:17.165932 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
I20240502 14:36:17.301919 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:17.304162 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:17.305306 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:17.312248 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:17.315609 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:17.336921 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:36:17.407413 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:36:17.481698 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:17.484087 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:17.527460 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:17.530474 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:17.540470 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:17.556396 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:17.559849 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:17.570940 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:36:17.578193 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20240502 14:36:17.622152 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20240502 14:36:17.623028 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
I20240502 14:36:17.770714 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:17.777158 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:17.779595 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:17.783248 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:17.800292 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:17.831013 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:36:17.873420 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:18.020763 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:18.043406 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:18.056497 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:18.059221 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:18.069590 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:36:18.071971 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:18.124917 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:18.150509 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 14:36:18.164304 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:36:18.262883 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:36:18.270821 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:18.282299 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:18.291712 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:18.292681 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:18.294201 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:36:18.310559 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:18.473042 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:18.491492 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:18.504688 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:18.510588 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:18.511500 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:18.512200 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:18.598757 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:18.710269 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:36:18.710712 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:36:18.767604 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:18.779335 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:18.783200 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:18.783571 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:18.786832 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:18.881412 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:36:18.956243 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:36:18.957464 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:18.973722 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:36:18.984719 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:18.985538 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:18.993440 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:18.994010 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:19.068521 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:19.111544 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:36:19.143976 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:36:19.154850 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:19.206624 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:19.207342 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:19.220455 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:19.232842 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:36:19.233573 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:19.236042 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:36:19.252306 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:19.256888 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 14:36:19.370290 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:19.429679 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:36:19.436450 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:19.440099 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:19.446264 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:19.448868 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:19.457463 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
I20240502 14:36:19.458043 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:19.542227 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:19.674763 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:19.683658 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:19.701879 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:19.707201 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:19.722035 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:19.731848 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:36:19.798353 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
I20240502 14:36:19.921530 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:19.931859 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:36:19.932744 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:19.939033 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:19.944059 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:19.952487 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:19.957496 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240502 14:36:19.959409 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20240502 14:36:19.967382 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:19.969820 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:20.148263 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:20.164451 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:20.166448 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:20.179272 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:20.180562 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:20.255195 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:36:20.339957 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20240502 14:36:20.360337 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
I20240502 14:36:20.370106 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:20.376341 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:20.382284 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:20.401110 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:20.401961 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:20.404029 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:20.497922 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:20.610798 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:20.626094 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:20.631438 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:20.632208 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:20.658756 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:20.713392 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:36:20.743853 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:36:20.804153 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:36:20.824869 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:36:20.866607 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:20.871822 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:20.897145 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:20.910998 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:20.911403 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:20.962986 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:20.987382 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:21.083204 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:21.085839 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:21.106194 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:21.109400 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:21.117393 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:21.238252 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:36:21.261754 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:36:21.320255 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:21.325210 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:21.332473 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:21.336289 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:21.340869 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:21.410094 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:36:21.485831 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:36:21.500083 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:21.514276 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:21.514545 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:21.532021 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:21.539225 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:21.545783 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:36:21.546439 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:21.572691 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:21.598254 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:36:21.606196 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:36:21.648862 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:36:21.733107 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:21.747680 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:21.749208 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:21.762070 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:21.769981 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:21.790293 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:36:21.871197 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
I20240502 14:36:21.955924 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:21.974596 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:21.975283 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:21.979012 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:21.979482 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:22.001096 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:36:22.039239 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:36:22.040450 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:22.106653 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:36:22.120126 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:22.175483 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:22.184440 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:22.200569 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:22.208420 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:22.208859 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:22.249504 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
I20240502 14:36:22.422852 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:22.430516 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:22.442170 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:22.447242 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:22.456447 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:22.466733 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:22.499883 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20240502 14:36:22.571868 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:36:22.592654 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:36:22.620002 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
I20240502 14:36:22.659440 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:22.659497 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:36:22.659984 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:22.682194 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:22.683878 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:22.685688 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:22.716468 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:22.744824 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
I20240502 14:36:22.909299 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:22.910902 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:22.920753 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:22.947010 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:22.947059 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:22.952514 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:23.150059 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:23.155664 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:23.161455 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:23.167060 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:23.181808 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:23.182426 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:36:23.203032 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 14:36:23.206331 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:36:23.276594 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:23.360193 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:23.368618 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:23.384398 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:36:23.385372 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:23.389586 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:23.394681 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:23.435530 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:36:23.490042 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:23.575924 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:23.584513 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:23.587844 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:23.609244 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:23.635958 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:23.698833 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:23.804512 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:36:23.854892 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:23.883666 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:23.888394 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:23.896392 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:23.905448 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:23.919309 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:23.923916 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:36:23.956076 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:36:24.006894 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:36:24.055810 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:36:24.081619 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:24.091647 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:24.093459 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:24.099309 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:36:24.113197 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:36:24.117415 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:24.128968 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:24.151147 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:24.277730 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:36:24.326153 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:24.336400 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:24.337007 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:24.342564 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 14:36:24.345728 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:24.355302 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:24.356072 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:24.437098 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20240502 14:36:24.468963 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:36:24.507154 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:36:24.580788 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:24.581526 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:24.593147 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:24.617259 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:24.620858 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:24.685973 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:36:24.714664 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:24.848088 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
I20240502 14:36:24.855567 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:24.861065 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:24.886582 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:24.906052 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:24.909034 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:24.923733 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20240502 14:36:24.954963 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:24.970187 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20240502 14:36:24.990010 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 98 similar messages]
I20240502 14:36:25.084587 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:25.094267 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:25.094928 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:25.117169 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:25.120517 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:25.158761 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:25.229102 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20240502 14:36:25.239683 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:36:25.294936 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:25.298767 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:25.304260 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:25.311493 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:36:25.312098 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:25.315618 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:25.445109 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:25.510980 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:25.518113 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:25.525154 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:25.527163 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:25.540804 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:25.545866 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:25.672915 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:36:25.749913 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:25.755836 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:25.767177 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:36:25.767794 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:25.770309 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:25.770645 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:25.825897 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:25.886924 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:36:25.974524 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:25.985709 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:25.988195 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:36:25.990330 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:25.992957 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:25.993532 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:25.998057 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:26.213860 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:26.240999 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:26.241358 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:26.242269 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:26.246573 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:36:26.249472 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:26.355358 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:26.417578 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:26.434599 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:36:26.444521 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:26.450413 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:26.451131 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:26.455003 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:26.458485 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:26.497685 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:36:26.510695 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:36:26.528918 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:36:26.529541 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:36:26.610181 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:36:26.696712 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:26.708170 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:26.717061 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:26.719489 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:26.725566 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:26.803534 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:26.852252 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:36:26.879228 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:36:26.880824 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
I20240502 14:36:26.918728 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:26.926730 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:26.927636 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:26.942397 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:26.956857 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:26.967754 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:27.017248 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:36:27.035744 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:36:27.063575 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 14:36:27.142210 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:27.143745 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:27.153769 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:27.165496 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:27.173166 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:27.332772 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:27.333165 17696 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:36:27.333406 17696 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:36:27.334286 17696 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
W20240502 14:36:27.337469 18637 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:36:27.338546 18636 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:27.338694 18636 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: 7ab6d395b30d4afb9821b61c9454430b, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:36:27.339540 17294 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:36:27.346860 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:27.360648 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:27.363850 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:27.374126 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:27.391839 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
I20240502 14:36:27.395126 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:27.455682 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:27.484591 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
I20240502 14:36:27.579545 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:27.592913 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:27.593886 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:27.598526 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
I20240502 14:36:27.608700 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:27.645457 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:27.745980 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:36:27.746500 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20240502 14:36:27.786942 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:36:27.829536 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:27.831893 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:27.845599 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:27.849727 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:27.863337 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:36:27.869786 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:27.899588 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:27.958911 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:28.033744 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:36:28.059641 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:28.064421 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:28.077867 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:28.078778 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:28.086256 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:28.278268 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240502 14:36:28.279464 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:28.290946 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:28.292510 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:28.300158 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:28.314744 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:28.331262 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:36:28.367183 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:28.493499 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:36:28.513253 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:28.515664 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:28.520520 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:28.533936 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:28.534294 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:28.542393 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:28.547993 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:36:28.760308 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:28.762703 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:28.770992 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:28.777037 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:28.783032 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:36:28.799088 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:28.898069 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:28.925652 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:36:29.011991 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:29.014029 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:29.031829 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:29.033295 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:29.053237 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:29.060048 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:36:29.091280 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:36:29.113695 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:29.122507 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:36:29.146484 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:36:29.199240 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20240502 14:36:29.223691 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:36:29.231418 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:36:29.249027 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:29.262274 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:29.263852 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:29.265278 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:29.265427 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:29.296255 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:29.330261 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:36:29.382699 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 14:36:29.465145 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:29.470438 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:29.470597 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:29.475055 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:29.478262 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:29.572302 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:36:29.600133 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:29.686453 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:36:29.710422 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:29.726373 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:29.728168 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:29.742264 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:29.749030 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:29.876821 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20240502 14:36:29.899091 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:29.920852 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:29.937083 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:29.937770 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:29.941494 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
I20240502 14:36:29.941991 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:29.967315 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:30.008579 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240502 14:36:30.010774 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
W20240502 14:36:30.134169 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:30.139041 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:30.151916 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:30.162230 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:30.168093 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:30.178339 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:30.188258 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20240502 14:36:30.299113 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:30.340688 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:36:30.366511 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:30.373410 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:36:30.379731 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:30.384364 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:30.388238 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:30.440122 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:30.585858 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:30.601140 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:30.605772 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:30.609762 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:30.622152 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:30.691522 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:30.718605 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:36:30.723114 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:30.760169 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:36:30.828372 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:30.831295 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:30.832592 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:30.847038 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:30.849567 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:30.882627 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:36:31.047510 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:31.054991 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:31.066545 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:31.080227 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:31.081511 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:31.094484 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:31.110881 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:36:31.136466 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:31.265674 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:31.272053 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:36:31.277104 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:31.280577 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:31.286353 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:31.298887 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:31.431983 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:36:31.475517 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:31.480706 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:31.488346 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
I20240502 14:36:31.490425 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:31.490901 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:36:31.493759 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:31.496881 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:31.500356 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:36:31.586479 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:31.635936 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:36:31.664597 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:36:31.674393 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:31.690762 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:31.700095 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:31.715019 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:31.737907 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:36:31.767925 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:31.768915 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:31.854018 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:36:31.942076 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:31.957669 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:31.971784 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:36:31.976437 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:31.978819 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:32.032910 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:32.106568 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:36:32.162503 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:32.163215 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:32.179327 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:32.183045 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 14:36:32.183775 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:32.184594 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:32.189005 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:32.224179 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:32.257447 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:36:32.292687 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20240502 14:36:32.384126 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
I20240502 14:36:32.416563 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:32.423782 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:32.451503 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:32.460278 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:32.460798 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:32.623711 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20240502 14:36:32.629719 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:32.631940 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:32.658841 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:32.666879 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:32.676882 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:32.701186 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:32.720520 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:32.738121 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:36:32.775122 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 14:36:32.781407 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20240502 14:36:32.884377 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:36:32.918155 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:32.918392 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:32.925463 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:32.930675 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:32.939446 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:32.949908 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:36:33.122973 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:33.123517 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:33.132388 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:33.136900 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:33.152369 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:33.208402 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:33.269795 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:33.306515 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:36:33.336141 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:33.363930 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:33.370023 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:33.370517 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:33.371706 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:33.385923 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 14:36:33.422237 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:36:33.444730 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:36:33.556494 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:33.560137 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:33.569629 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:33.572371 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:33.585783 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:33.687031 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:36:33.769640 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:33.801827 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:33.811209 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:33.811635 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:33.828579 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:33.830682 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:33.838253 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:33.856776 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:36:33.937048 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:36:34.027683 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
I20240502 14:36:34.075335 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:34.075739 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:34.082873 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:34.087980 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:34.104434 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:34.183396 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:36:34.220830 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:36:34.248755 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:36:34.254805 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:36:34.280452 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:36:34.284968 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:34.288211 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:34.304013 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:34.307864 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:34.310545 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:34.313755 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:34.385515 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 14:36:34.412813 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:34.476254 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:34.486644 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:34.490170 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:34.498682 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:34.507771 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:34.527698 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:36:34.632683 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20240502 14:36:34.709182 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:34.731204 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:36:34.748342 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:34.756807 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:34.761687 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:34.762374 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:34.767935 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:34.804961 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20240502 14:36:34.918274 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:34.940879 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:34.955063 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:34.963318 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:34.965397 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:34.986302 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:35.012161 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240502 14:36:35.119307 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:36:35.150964 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
I20240502 14:36:35.162263 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:35.166426 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:36:35.173732 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:35.174412 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:35.175153 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:35.185981 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:35.208639 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:35.285259 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
I20240502 14:36:35.372661 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:35.377435 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:35.387782 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:35.388401 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:35.406484 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:35.497622 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:36:35.500646 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:35.566855 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:36:35.567571 18544 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 14:36:35.572372 18544 catalog_manager.cc:4573] TS a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 246a8425400f4890a5264c4ae8a627c5: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:35.612941 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:35.614133 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:35.625680 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:35.629268 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:35.633678 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:35.757957 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:35.833979 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:35.834566 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:35.861590 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:35.863243 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:35.873848 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:35.936250 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:35.962194 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:36:36.032657 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:36:36.055902 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:36.063197 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:36:36.063894 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:36.069840 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:36.073660 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:36.085577 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:36.122385 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:36:36.261541 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
I20240502 14:36:36.294953 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:36.298939 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:36.303725 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:36.305967 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:36.307078 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:36.318318 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:36.324115 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:36.340972 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:36:36.507865 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:36.514279 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:36.525679 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:36.526645 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:36.537714 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:36.582717 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:36:36.686218 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:36:36.698899 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:36.740064 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:36:36.773644 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:36.778919 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:36.789944 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:36.797951 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:36.801631 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:36.810494 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:36:36.818662 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:36:36.824606 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:36:36.865458 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:36:36.876570 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:36.929217 18544 catalog_manager.cc:4573] TS a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6a992d12aee7416096d1353533397386: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:37.022540 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:37.038235 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:37.044499 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:37.066351 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:37.068342 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:37.113762 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:37.144517 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20240502 14:36:37.206377 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:36:37.255926 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
I20240502 14:36:37.263761 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:37.271576 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:37.274704 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:37.276526 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:37.299971 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:37.418196 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:36:37.419126 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:37.437283 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
I20240502 14:36:37.465826 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:37.468472 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:37.475699 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:37.485457 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:37.492681 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:37.597604 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:37.686322 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:37.725366 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:37.731001 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:37.731796 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:37.764286 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:37.814558 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:36:37.880144 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:36:37.902297 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:36:37.947726 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:37.958129 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:37.959182 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:37.959538 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:37.962983 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:37.966274 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:36:38.012490 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:38.016062 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:38.156004 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:38.179785 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 14:36:38.183400 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:38.186977 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:38.194756 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:38.204470 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
I20240502 14:36:38.219667 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:38.401118 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:38.403441 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:38.414994 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:38.436060 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:38.449982 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:38.483857 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:36:38.490967 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 14:36:38.504985 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:38.594981 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:36:38.596645 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:38.609911 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:36:38.640540 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:38.643306 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:38.655207 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:36:38.661242 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:38.661981 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:38.665674 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:38.739619 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
I20240502 14:36:38.905277 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:38.914417 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:38.925912 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:38.931663 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:38.946040 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:39.023701 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:39.098249 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:36:39.099727 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:39.104565 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:36:39.131989 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:39.134757 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:36:39.146153 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:39.153131 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:39.162259 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:39.171871 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:36:39.193284 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:39.250015 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:36:39.357362 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:36:39.366470 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:39.368949 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:39.388712 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:39.401281 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:39.405664 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:39.417564 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:36:39.430680 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:36:39.440651 18544 catalog_manager.cc:4573] TS a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 0ffb6e582ef04fb38dd273c13febac58: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:39.478255 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:36:39.480758 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:39.579474 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:39.579663 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:39.589614 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:39.591565 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:39.594983 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:39.608629 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:39.622380 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20240502 14:36:39.785715 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
I20240502 14:36:39.842402 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:39.846261 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:39.849496 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:39.849826 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:39.878891 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:39.905581 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:36:39.963232 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:40.040910 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:40.043433 18768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 98 similar messages]
I20240502 14:36:40.046013 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:40.057051 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:40.073935 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:40.076148 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:40.090862 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:40.134595 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20240502 14:36:40.223188 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:36:40.275215 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:36:40.297462 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:40.303102 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:40.306896 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:40.317348 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:40.319902 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:40.354475 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:40.461598 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:36:40.472535 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
W20240502 14:36:40.481786 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:36:40.500315 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:40.513698 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:40.513834 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:40.530385 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:40.537988 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:40.547516 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:40.758349 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:40.772193 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:40.773012 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:40.801240 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:40.802388 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:40.804692 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:36:40.910945 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:40.972918 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:36:40.992698 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:40.995878 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:40.998915 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:41.003576 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:41.004249 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:41.072816 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:41.141793 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:36:41.200090 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:41.209348 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:41.214136 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:41.227419 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:41.246101 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:41.298564 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:36:41.341873 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20240502 14:36:41.349751 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:41.423465 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:36:41.436887 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:41.454030 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:41.454861 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:41.466145 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:41.466888 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:41.549067 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:36:41.577659 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:41.665829 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:41.675433 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:41.679004 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:41.687196 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:41.705813 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:41.758266 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:36:41.775444 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:41.857605 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:36:41.889091 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:36:41.896135 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:36:41.949898 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:41.950213 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:41.961513 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:36:41.969576 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:41.972663 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:41.988353 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:36:41.992247 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:42.177924 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:42.184808 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:42.188578 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:42.214262 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:42.219589 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:42.221118 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:42.227900 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
I20240502 14:36:42.237376 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:42.359015 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
I20240502 14:36:42.439551 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:42.440434 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:42.461345 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:42.461552 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:42.461751 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:42.495484 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 14:36:42.545327 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:36:42.563172 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20240502 14:36:42.593601 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:42.618863 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:36:42.660480 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:36:42.709726 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:42.716356 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:42.716846 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:42.718945 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:42.720628 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:42.741221 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:42.754842 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:36:42.925046 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:42.933936 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:36:42.934837 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:42.941941 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:42.955200 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:42.973682 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:43.172304 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
W20240502 14:36:43.172814 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:43.190584 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:43.190919 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:43.195433 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:43.215454 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:43.221946 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:43.233947 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:36:43.302292 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:43.399443 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:36:43.405465 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:43.410732 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:43.411690 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:43.418924 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:43.427544 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:43.504616 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240502 14:36:43.506417 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:36:43.626577 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:43.651218 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:43.667366 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:43.685649 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:43.698415 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:43.702487 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:43.775661 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:36:43.777547 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:36:43.805088 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:36:43.900655 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:43.913321 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:36:43.922091 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:43.930545 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:43.936192 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:43.936945 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:43.953289 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:44.016294 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20240502 14:36:44.087718 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:44.152490 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:44.164098 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:36:44.165653 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:44.170087 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:44.187959 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:44.189114 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:44.207018 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:36:44.292624 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:36:44.314019 21224 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:36:44.314215 21224 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:36:44.315117 21224 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
W20240502 14:36:44.316437 18636 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:44.316707 18637 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:36:44.316824 18637 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: 7ab6d395b30d4afb9821b61c9454430b, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:36:44.317044 21224 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:36:44.336669 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:36:44.378708 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:44.381486 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:44.395314 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:44.400951 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:44.406314 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:44.489238 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:44.491667 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:44.540019 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:36:44.545040 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:36:44.546648 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
I20240502 14:36:44.582913 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:44.583164 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:44.593983 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:44.613656 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:44.627732 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:44.703290 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20240502 14:36:44.731292 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:36:44.790633 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:36:44.829140 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:44.845851 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:44.852973 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:44.853148 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:44.871207 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:44.911608 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20240502 14:36:44.988493 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:45.046739 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:45.062393 18768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 101 similar messages]
I20240502 14:36:45.076339 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:45.084103 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:36:45.084604 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:45.094178 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:45.097931 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:45.120874 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:45.142901 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:36:45.225098 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:36:45.300287 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:45.301029 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:45.315984 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:45.323637 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:45.333889 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:45.524024 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:45.534283 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:45.548640 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:45.568027 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:45.572553 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:45.612563 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:45.638250 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:36:45.657204 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:45.695469 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:36:45.777767 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:45.783910 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:45.793831 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:45.794657 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:45.813320 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:45.917819 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20240502 14:36:45.973418 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:36:46.004694 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:46.011008 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:46.021452 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:46.025746 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:46.025957 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:46.029007 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:46.069301 18769 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.67:45409, user_credentials={real_user=slave}} blocked reactor thread for 44962.5us
W20240502 14:36:46.111552 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:36:46.217542 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:46.226580 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:46.237469 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:46.243867 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:46.261940 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:46.284821 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:46.315076 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:36:46.328727 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:36:46.390352 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:36:46.460883 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:46.463760 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:46.475514 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:46.479672 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:46.482120 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:46.501335 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:46.501611 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:36:46.584606 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20240502 14:36:46.654130 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:36:46.713057 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:36:46.739097 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:46.748317 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:46.750939 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:46.779327 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:46.781810 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:46.805796 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:36:46.819653 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:46.819890 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:36:46.954295 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
I20240502 14:36:46.964352 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:46.982653 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:46.985836 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:46.986188 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:46.986867 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:47.079633 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:47.152169 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:36:47.198976 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:47.212515 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:47.212909 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:47.213528 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:47.225803 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:47.258872 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
I20240502 14:36:47.299012 21755 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:36:47.299224 21755 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:36:47.300083 21755 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
W20240502 14:36:47.304862 18637 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:36:47.305199 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:36:47.307190 18636 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:47.307318 18636 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: 7ab6d395b30d4afb9821b61c9454430b, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:36:47.307893 21755 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:36:47.408715 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20240502 14:36:47.424994 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:47.429169 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:47.435526 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:47.435523 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:36:47.439976 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:47.449725 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:47.503165 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:47.539149 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 14:36:47.549402 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:36:47.559614 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:47.618691 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:47.644845 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:47.670790 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:47.675045 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:47.676680 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:47.832015 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:36:47.903649 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:47.905061 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:47.907368 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:47.925602 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:47.927029 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:48.025388 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:48.128821 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:48.141916 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:48.143501 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:48.146365 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:48.153975 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:48.172540 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:48.232685 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:36:48.320480 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:36:48.380475 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:48.380877 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:48.387989 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:48.389279 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:48.390221 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
I20240502 14:36:48.406342 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:48.421563 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:48.465679 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 14:36:48.536140 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:36:48.576090 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 65 similar messages]
I20240502 14:36:48.604516 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:48.614223 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:48.629374 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:48.635892 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:48.638880 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:48.689127 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:48.731410 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:36:48.819527 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:36:48.825492 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:48.849448 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:48.860993 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:48.881693 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:48.884380 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:36:48.902666 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:48.931324 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:49.011655 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:36:49.068136 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:36:49.086515 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:36:49.101588 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:36:49.111052 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
I20240502 14:36:49.126840 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:49.132454 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:49.134745 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:49.136732 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:49.145713 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:49.227851 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:49.233744 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:36:49.298003 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:49.344970 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:49.347364 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:49.353979 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:49.360766 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:49.374233 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:49.386127 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20240502 14:36:49.445775 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:36:49.488996 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:36:49.500005 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:36:49.541939 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:49.550568 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:49.552042 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:49.567865 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:49.586524 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:49.615759 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:49.699982 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20240502 14:36:49.753369 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 14:36:49.787453 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:49.796181 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:49.816278 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:49.819823 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:49.820552 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:49.863360 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:49.954500 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
I20240502 14:36:50.016727 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:50.019371 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:50.022503 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:50.034279 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:36:50.034865 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:50.037545 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:50.098551 18768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240502 14:36:50.149722 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:36:50.226270 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:50.234856 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:50.246412 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:50.248986 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:50.268637 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:50.294256 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:50.367218 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:36:50.419427 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:50.441083 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:50.449447 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:50.457785 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:50.474707 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:50.477349 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:50.698760 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:50.703969 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:50.722332 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:50.727411 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:50.733405 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:50.733765 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:50.796073 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:36:50.897009 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20240502 14:36:50.914186 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:50.937330 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:50.949479 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:50.954385 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:50.960546 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:50.994621 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:36:50.999924 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:51.127233 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:36:51.179895 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:51.190974 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:51.194360 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:36:51.194651 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:51.208428 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:51.222460 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:51.225199 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:36:51.265146 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:51.280082 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:36:51.377506 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:36:51.409868 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:51.419636 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:51.419950 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:51.429838 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:51.440188 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:51.515556 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20240502 14:36:51.529917 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:51.615470 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:36:51.647284 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:51.663187 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:51.676028 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:51.710608 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:51.730000 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:51.731576 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:36:51.786691 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:36:51.795348 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:36:51.885843 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:51.888625 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:51.914572 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:51.915282 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:51.919961 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:51.920787 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:51.929234 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20240502 14:36:52.006253 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:36:52.036485 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:52.059830 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
I20240502 14:36:52.096160 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:52.109916 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:52.110459 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:52.113658 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:52.125690 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:52.142441 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20240502 14:36:52.245240 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:36:52.267496 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:36:52.322199 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:52.330665 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:52.330979 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:52.343312 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:52.348435 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:52.445111 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:52.481402 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:52.521344 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:52.528695 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:52.537986 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:52.538705 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:52.549451 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:52.564245 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 14:36:52.637246 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:36:52.756249 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:52.766803 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:52.785578 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:52.786329 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:52.795440 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:52.825503 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:36:52.893268 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:52.951223 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:36:52.983737 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:53.007922 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:53.015095 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:53.034760 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:53.059700 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:53.085716 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:53.226840 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
I20240502 14:36:53.282253 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:53.283330 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:53.284965 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:53.285640 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:53.344916 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:53.352919 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:36:53.370093 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:53.468930 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:53.481870 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:36:53.488461 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:53.489369 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:53.490406 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:53.492040 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:53.586844 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:53.590411 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:36:53.591154 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:36:53.661839 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:36:53.693154 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240502 14:36:53.746484 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:53.755611 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:53.761830 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:53.782104 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:53.783226 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:53.810920 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:36:53.826701 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:36:53.897544 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:54.001766 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:54.012656 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:54.018402 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:54.025722 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:54.032446 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:54.056485 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20240502 14:36:54.142235 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:54.208943 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:54.216125 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:54.221979 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:54.234336 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:54.247323 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:36:54.259066 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:54.261981 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:36:54.283411 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:54.297510 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20240502 14:36:54.299268 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20240502 14:36:54.310698 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:36:54.343726 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:36:54.411751 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:36:54.437516 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:54.439504 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:54.443404 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:54.453174 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:54.461863 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:54.492897 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:36:54.503182 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20240502 14:36:54.571496 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:36:54.669615 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:54.683018 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:54.691059 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:54.693070 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:54.720391 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:54.724500 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:54.747341 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:54.786825 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:36:54.886698 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 14:36:54.909087 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:54.910737 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:54.923610 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:54.929106 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:54.960477 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:55.145344 18769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 99 similar messages]
I20240502 14:36:55.150172 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:55.158856 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:55.164364 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:55.167129 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:55.178961 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:55.179509 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:55.216282 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:36:55.245735 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:36:55.263783 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:36:55.296116 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:55.372278 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:55.382553 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:55.383857 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:55.384238 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:55.400751 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:55.568948 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:55.580938 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:55.597970 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:55.600062 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:55.624608 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:55.625465 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:55.725239 23326 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:36:55.725399 23326 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:36:55.726025 23326 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
W20240502 14:36:55.727531 18636 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:55.729983 18637 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:36:55.730130 18637 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: 7ab6d395b30d4afb9821b61c9454430b, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:36:55.730352 23326 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:36:55.771745 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:36:55.804244 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:36:55.828326 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:55.834666 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:55.849809 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:55.863415 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:55.874305 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:55.878087 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:55.935544 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20240502 14:36:55.956533 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:56.073871 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:56.088610 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:56.110858 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:36:56.133778 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:56.145924 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:56.148752 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:56.244490 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:36:56.288652 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:36:56.298645 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:56.303711 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:56.312322 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:56.314689 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
I20240502 14:36:56.315721 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:56.332864 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:56.337021 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:36:56.413165 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:56.529345 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:56.547755 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:56.550097 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:56.551970 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:56.553944 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:56.561514 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:56.760468 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:36:56.782989 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:56.783437 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:56.802484 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:56.808233 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:56.808588 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:36:56.809628 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
I20240502 14:36:56.811442 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:56.822578 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:56.982657 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:57.000380 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:57.000411 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:57.002972 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
I20240502 14:36:57.005489 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:57.008621 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:36:57.021400 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:36:57.033998 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:36:57.034283 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
I20240502 14:36:57.038671 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:57.056229 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:36:57.057091 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:57.094460 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:36:57.158867 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:36:57.218984 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:57.230371 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:57.241896 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:57.254894 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:57.277556 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:57.280059 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:57.445148 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:57.446767 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:57.461617 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:57.473409 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:57.496976 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:57.520164 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:57.559408 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:36:57.584501 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 27 similar messages]
I20240502 14:36:57.689764 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:57.707302 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:57.711061 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:57.714093 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:57.731065 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:57.749487 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:36:57.863955 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:57.949921 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:57.954342 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:57.956864 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:57.960613 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:57.966468 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:57.978042 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:57.984730 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:36:58.168115 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:58.172391 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:58.188634 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:58.195616 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:58.198151 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:58.287092 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:36:58.388494 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:58.389147 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:58.396598 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:36:58.406772 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:58.418709 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:58.419400 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:58.424016 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20240502 14:36:58.454823 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:58.593840 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:58.594694 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 18 similar messages]
I20240502 14:36:58.604615 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:58.630620 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:36:58.641772 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:36:58.642212 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:58.646076 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:58.667181 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:36:58.739120 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240502 14:36:58.853004 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:36:58.857841 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:58.864238 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:58.868081 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:58.874083 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:58.879326 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:36:58.929739 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:36:58.936688 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:36:58.957249 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20240502 14:36:58.966149 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:58.967981 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:36:59.082712 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:59.083694 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:59.096585 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:59.102548 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:59.118264 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:59.267549 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:36:59.272357 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
I20240502 14:36:59.305568 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:59.306278 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:59.317117 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:59.322360 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:59.337859 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:59.383276 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:36:59.448705 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:36:59.454589 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:36:59.481537 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:36:59.486781 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:36:59.534852 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:59.536861 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:59.542366 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:59.544766 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:36:59.549108 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:59.558192 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:36:59.579255 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:36:59.627575 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20240502 14:36:59.729118 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
I20240502 14:36:59.761059 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:59.767032 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:36:59.769630 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:36:59.775278 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:59.778245 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:36:59.793401 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:36:59.883538 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:36:59.919309 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240502 14:36:59.981243 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:36:59.988238 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:36:59.991717 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:00.008065 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:00.009809 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:00.068483 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:00.140201 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:00.165450 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:37:00.182404 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:00.187557 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:00.214411 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:00.229061 18768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 99 similar messages]
I20240502 14:37:00.234092 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:00.237839 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:00.422652 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:37:00.429481 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:00.438438 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:00.447438 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:00.451402 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:00.467358 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:00.495436 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:37:00.496477 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:00.657428 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:00.665484 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:00.666170 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:00.673801 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:00.685874 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:00.692629 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:37:00.892719 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:00.898262 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:00.906914 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:00.915977 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:00.926436 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:00.950910 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:37:01.020363 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
W20240502 14:37:01.067193 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:01.083643 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:37:01.130859 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:01.135778 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:01.144146 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:37:01.147855 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:01.153123 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:01.158039 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:37:01.177107 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:01.351012 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:37:01.358841 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:01.373561 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:01.376416 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:37:01.380034 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:37:01.383713 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:01.417750 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:01.577473 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:37:01.577616 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:37:01.610553 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:37:01.611112 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
I20240502 14:37:01.615509 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:01.622174 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:01.624018 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:01.626386 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:01.643464 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:01.673606 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:37:01.759362 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:01.833056 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20240502 14:37:01.833698 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:37:01.859529 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:01.871686 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:01.885078 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:01.885772 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:01.902376 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:01.922839 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:37:01.989895 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:37:02.034229 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20240502 14:37:02.068238 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:02.085227 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:02.088243 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:02.106321 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:37:02.110376 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:02.112131 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:02.142156 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:02.268996 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:37:02.272413 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20240502 14:37:02.307050 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:02.308218 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:37:02.348134 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:02.350329 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:02.366658 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:02.372944 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:02.376158 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:02.525949 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:02.551648 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:02.566282 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:02.569782 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:02.573781 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:02.574509 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:02.678021 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:37:02.701501 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:37:02.811003 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240502 14:37:02.826653 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:02.831979 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:02.841408 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:02.854787 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:02.858940 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:02.887403 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:37:03.026007 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:03.028596 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:03.040621 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:03.045023 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:03.063189 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:03.131336 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:03.187072 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:37:03.243347 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:37:03.283792 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:03.290545 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:03.299074 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:03.303618 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:03.307111 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:03.363787 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:03.462687 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:37:03.473824 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:03.474565 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:03.490060 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:03.497815 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:03.503221 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:03.546367 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:03.617669 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:37:03.620270 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
I20240502 14:37:03.738437 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:03.744282 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:03.745564 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:37:03.752661 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:03.761644 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:03.765380 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:03.777667 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 14:37:03.789148 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:03.843851 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 14:37:03.844884 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:37:03.973587 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:03.978899 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:03.979610 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:03.987514 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:04.003993 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:04.004109 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:04.025099 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:37:04.106755 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:37:04.168804 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:04.184291 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:37:04.216233 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:04.219272 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
I20240502 14:37:04.226572 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:37:04.242199 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:37:04.245647 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:04.248216 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:04.250685 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:04.279430 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20240502 14:37:04.279765 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:37:04.388917 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:37:04.439821 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:04.450747 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:04.453343 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:04.458794 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:04.476148 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:04.500710 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:04.504660 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:37:04.574353 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:37:04.651340 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20240502 14:37:04.660526 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20240502 14:37:04.664366 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:37:04.682134 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:04.686089 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:04.715003 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:04.773311 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:04.773660 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:04.822707 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:37:04.951959 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:04.977388 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:04.979173 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:04.980607 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:05.002463 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:05.030463 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:37:05.058990 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:05.059284 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:37:05.105562 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:37:05.121870 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:05.179118 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:05.185292 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:05.191731 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:05.192027 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:05.195515 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:05.267083 18768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 102 similar messages]
I20240502 14:37:05.393692 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:05.407946 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:05.421231 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:05.426417 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:05.426827 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:05.557997 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:37:05.589417 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:05.611936 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:05.613960 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:05.622145 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:05.623189 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:05.632920 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:37:05.666759 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:05.799472 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:37:05.825553 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:05.836707 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:05.840014 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:05.840790 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:05.843014 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:05.988271 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:37:06.091389 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:06.094094 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:06.098126 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:06.099139 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:37:06.100175 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:06.104697 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:06.132905 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:37:06.167064 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:37:06.177455 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20240502 14:37:06.245455 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:06.286168 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:06.295621 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:06.305238 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:06.309077 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:06.312582 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:06.325937 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20240502 14:37:06.405790 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:06.440630 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:37:06.441747 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:37:06.495116 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:06.501605 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:06.504549 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:06.518257 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:06.526436 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:06.535296 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:37:06.622469 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:06.681871 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:37:06.694936 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:37:06.705210 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:37:06.744001 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:06.746488 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:06.759742 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:06.769299 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:06.806195 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:06.858194 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:06.886946 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20240502 14:37:06.904407 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:37:06.917471 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:37:06.968333 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:37:06.994958 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:37:07.007647 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:07.009079 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:07.010182 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:07.012712 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:07.052875 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:07.075592 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20240502 14:37:07.078840 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20240502 14:37:07.092339 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:07.239449 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:07.245325 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:07.266443 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:07.268522 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:07.297744 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:07.355729 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:37:07.372879 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:37:07.483230 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:07.483508 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:07.493785 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:07.503183 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:07.504961 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:07.551357 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:07.599359 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:37:07.744443 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:07.744738 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:07.756664 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:07.757911 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:07.760561 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:07.872972 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240502 14:37:07.874069 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:07.926481 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:37:07.946543 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:07.953383 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:07.956205 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:07.958511 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:07.976473 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:08.037770 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:08.183367 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:08.193058 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:08.202983 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:08.203083 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:08.216504 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:08.311003 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:37:08.403198 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:08.415820 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:08.421465 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:08.424552 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:08.430099 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:08.435611 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:37:08.439090 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:08.638289 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:08.640053 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:08.646301 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:08.670837 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:08.673346 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:08.703115 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:08.758461 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:37:08.775938 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:37:08.813971 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:08.840590 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20240502 14:37:08.863265 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 14:37:08.864626 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:37:08.872438 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:08.872987 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:08.905872 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:08.911811 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:08.912401 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:08.959275 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:37:09.014397 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20240502 14:37:09.040069 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:37:09.114070 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:09.121086 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:09.121477 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:09.127501 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:09.135197 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:09.141309 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:37:09.150243 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:09.165881 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:37:09.229055 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:09.350203 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
I20240502 14:37:09.372874 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:09.378623 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:37:09.387636 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:09.396804 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:09.400878 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
I20240502 14:37:09.403167 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:09.420550 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:09.440850 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:37:09.501617 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:37:09.502761 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:37:09.548650 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20240502 14:37:09.567729 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:09.579430 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:09.580845 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:37:09.583526 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:37:09.601604 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:09.618081 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:09.660948 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:09.673839 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:37:09.814918 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:37:09.837848 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:37:09.838016 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:09.845952 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:09.849619 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:09.853755 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:09.868973 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:09.940135 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:37:10.062482 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:10.075263 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:10.087239 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:10.097317 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:10.111260 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:10.192095 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:10.249595 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:37:10.294657 18768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 100 similar messages]
I20240502 14:37:10.295464 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:10.309384 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:10.314702 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:10.316375 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:10.316437 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:10.358597 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:10.480072 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:37:10.501991 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:10.511000 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:10.524510 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:10.525349 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:10.540829 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:10.666936 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:10.749665 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:37:10.769737 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:10.775841 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:10.775846 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:10.777376 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:10.793354 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:10.885490 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:37:11.002719 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:11.003405 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:11.016311 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:11.025533 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:11.026603 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:11.103771 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:11.122902 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20240502 14:37:11.125614 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:37:11.225040 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:11.229028 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:11.235142 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:11.245158 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:11.247884 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:11.306315 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:37:11.323681 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:11.414312 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:37:11.425503 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
I20240502 14:37:11.433259 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:11.437450 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:11.444217 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:11.453434 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:11.455562 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:11.539281 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:37:11.666838 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:37:11.674458 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:37:11.694376 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:11.696039 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:37:11.710618 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:11.711769 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:11.712400 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:11.713959 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:37:11.714708 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:11.727818 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:11.772111 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:37:11.808097 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20240502 14:37:11.896865 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
I20240502 14:37:11.925060 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:11.936240 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:11.941143 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:11.948671 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:11.971487 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:37:11.972585 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:12.004539 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20240502 14:37:12.018942 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:37:12.142724 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:12.158118 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:12.159191 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:12.164292 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:12.175253 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:12.186918 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:12.208259 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:12.223446 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:37:12.223757 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:37:12.302893 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:37:12.373463 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:12.376866 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:12.408329 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:12.423329 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:12.430397 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:12.455129 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:37:12.586104 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:37:12.603766 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:12.629410 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:12.640940 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:37:12.645493 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:12.664914 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:37:12.680573 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:12.707077 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:12.892747 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:12.893644 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:12.915414 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:12.948104 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:12.957844 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:13.019237 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:37:13.073406 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:37:13.113905 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:13.126448 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:13.149811 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:13.150995 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:13.153837 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:13.245958 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:13.263724 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:37:13.339195 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:13.356523 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:13.357991 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:13.362654 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:13.377563 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:13.441558 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:37:13.562948 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:13.569710 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:13.581096 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:13.586794 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:13.597467 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:13.665680 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:13.706017 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:13.707664 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:37:13.712438 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:37:13.726127 26595 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:37:13.726295 26595 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:37:13.727048 26595 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
W20240502 14:37:13.728628 18636 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:13.728756 18637 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:37:13.728874 18637 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: 7ab6d395b30d4afb9821b61c9454430b, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:37:13.729287 26595 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:37:13.769001 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
I20240502 14:37:13.786825 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:13.797546 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:13.812803 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:13.818773 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:13.832258 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:13.894490 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20240502 14:37:13.949282 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 14:37:13.950259 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:37:13.956328 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:37:14.005836 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:14.020470 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:14.025267 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:14.028915 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:14.043241 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:14.146098 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:37:14.181466 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:37:14.189492 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:37:14.208261 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:37:14.218719 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:14.270385 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:14.271751 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:14.277428 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:14.279562 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:14.283007 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:14.285919 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:14.366855 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20240502 14:37:14.404917 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:37:14.438612 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:37:14.455693 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
I20240502 14:37:14.485281 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:14.493934 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:14.494624 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:14.494868 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:14.511950 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:14.538678 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:37:14.554167 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20240502 14:37:14.570168 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:37:14.671996 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:37:14.706713 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:14.715067 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:14.717064 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:14.719923 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:14.731066 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:14.832211 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:14.845808 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:37:14.862268 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:14.923774 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:14.945775 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:14.947005 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:14.955683 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:14.961045 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:15.039361 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:37:15.177172 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:15.200847 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:15.218811 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:15.219605 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:37:15.220402 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:15.225327 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:15.235677 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:37:15.309710 18766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240502 14:37:15.310708 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:15.342366 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 14:37:15.433272 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:15.442219 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:15.445019 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:15.449479 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:37:15.451659 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:15.462684 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:15.655485 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:15.656993 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:15.693298 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:15.693392 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:15.711325 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:15.729399 18900 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.107.69:41119, user_credentials={real_user=slave}} blocked reactor thread for 57321.8us
W20240502 14:37:15.799247 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:15.858389 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:37:15.906989 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:37:15.956175 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:15.957805 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:15.967339 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:15.970281 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:15.987854 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:16.021193 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:37:16.172834 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:16.179244 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:16.181094 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:16.181416 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:16.189736 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:16.274796 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20240502 14:37:16.355415 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:16.376082 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:16.385234 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:37:16.396672 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:37:16.414068 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:16.426293 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:16.434273 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:16.456264 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:16.477051 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
I20240502 14:37:16.477546 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:16.625707 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:16.650398 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:37:16.659955 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:37:16.679737 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20240502 14:37:16.706517 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:37:16.725243 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:16.728494 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
I20240502 14:37:16.733354 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:16.743404 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:16.748174 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:16.813841 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:37:16.863760 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:16.905587 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:37:16.914170 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:16.918630 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:16.928431 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:16.948359 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:16.951154 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:16.969305 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:37:16.998461 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:37:17.070930 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:37:17.119539 27135 raft_consensus.cc:482] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff6acba046404ef0a6f41ad5e09d852f)
I20240502 14:37:17.119721 27135 raft_consensus.cc:504] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "7ab6d395b30d4afb9821b61c9454430b" member_type: VOTER last_known_addr { host: "127.5.107.71" port: 39195 } }
I20240502 14:37:17.120440 27135 leader_election.cc:288] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195)
W20240502 14:37:17.122334 18637 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
W20240502 14:37:17.126364 18636 leader_election.cc:334] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:37:17.126482 18636 leader_election.cc:302] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: 7ab6d395b30d4afb9821b61c9454430b, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:37:17.126955 27135 raft_consensus.cc:2726] T 1ecc6539b85d40e49df7ffc5f93cb801 P 2db304b538e14857b7b1cc272d0db94b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:37:17.142313 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:17.145124 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:17.146689 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:17.149816 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:17.151329 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:17.169054 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:37:17.178406 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:37:17.191699 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:37:17.213346 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:17.341291 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:37:17.347546 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:17.352828 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:17.357883 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:17.363301 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
I20240502 14:37:17.371083 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:17.380236 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:17.535753 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:37:17.598644 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:37:17.600009 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:17.602105 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:17.604204 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:17.632771 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:17.637887 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:17.730419 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:37:17.848320 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:37:17.855970 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:37:17.857393 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:17.865680 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:17.879781 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:17.913123 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:17.929174 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:37:18.067886 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:18.084712 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 14:37:18.086506 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:18.110560 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:37:18.115685 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:37:18.116359 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:18.117400 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:18.119194 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:18.286847 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:18.293782 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:18.318089 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:18.337961 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:18.365777 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:18.521505 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:18.524434 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:18.533851 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:18.545047 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:18.545079 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:18.548945 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:18.555644 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20240502 14:37:18.645458 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:37:18.668349 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:37:18.686931 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:18.763615 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
I20240502 14:37:18.795641 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:18.801466 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:18.803251 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:18.804565 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:18.830237 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:18.844966 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:37:18.851267 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:37:18.956135 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:37:19.011288 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 14:37:19.025945 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:19.044605 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:19.056411 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:19.069423 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:19.070911 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:19.080004 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:19.097440 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:37:19.121928 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20240502 14:37:19.175369 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:19.224172 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:37:19.258790 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 14:37:19.290815 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:19.306550 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:19.320849 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:19.323746 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:37:19.343030 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:19.345614 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:37:19.351652 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:19.403224 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20240502 14:37:19.448014 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:37:19.484887 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:19.506462 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:19.517666 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:19.519963 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:37:19.528739 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:19.549091 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:19.550205 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:19.686481 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:37:19.690331 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:19.758111 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:37:19.763845 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:19.777470 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:19.778283 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:19.778482 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:19.798139 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:19.838246 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:37:19.911690 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:37:19.946839 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20240502 14:37:19.957269 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:20.000500 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:20.018815 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:20.026322 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:20.038970 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:20.076107 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:20.226482 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:20.239388 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:20.247889 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:20.264411 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:20.279080 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:37:20.294945 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:37:20.313241 18768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 97 similar messages]
W20240502 14:37:20.364590 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 14:37:20.408823 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:37:20.446763 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:37:20.454522 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:20.454617 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:20.468797 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:20.470875 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:20.473343 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:20.513660 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:20.682857 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:20.692651 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:20.715885 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:20.720090 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:20.727952 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:20.839028 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:37:20.943157 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:20.948211 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:20.958534 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:20.959103 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:20.965199 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:21.003767 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20240502 14:37:21.083549 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:21.142395 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:37:21.165062 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
I20240502 14:37:21.189563 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:21.191015 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:21.195439 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:21.198540 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:21.201704 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:21.212245 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:37:21.237058 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:37:21.259750 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:37:21.346913 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:21.361842 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:37:21.366415 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:37:21.375398 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
I20240502 14:37:21.396522 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:21.418989 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:21.420325 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:21.425597 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:21.434327 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:21.520188 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:37:21.628866 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:21.633000 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:37:21.638332 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:21.641655 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:21.642604 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:21.681274 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:21.815080 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:37:21.875530 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:21.884200 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:37:21.886024 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:37:21.889472 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:21.896121 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:21.901109 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:21.921376 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:21.968989 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:22.072446 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:37:22.099037 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:22.120013 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:22.128466 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:22.128607 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:22.156630 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:37:22.160246 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:37:22.175814 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:22.180836 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:37:22.230399 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20240502 14:37:22.248410 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
I20240502 14:37:22.354552 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:22.373893 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:22.376447 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:22.380152 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:22.417899 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:22.517213 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:37:22.584748 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:22.591034 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:22.591485 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:22.597504 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:37:22.609109 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:37:22.609436 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:22.611317 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:22.704692 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:22.758203 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:37:22.826861 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:22.838596 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:22.842191 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:22.859751 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:22.864329 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:22.940268 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:37:23.039602 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:23.045114 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:23.065279 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:23.070534 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:23.088528 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:37:23.105624 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:23.119172 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:37:23.205051 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:23.277802 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:23.291667 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:23.292608 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:23.298314 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:23.300527 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:23.492347 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:23.510375 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:37:23.515267 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:23.516242 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:23.524003 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:23.546069 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:23.554970 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:37:23.652616 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:37:23.691124 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:37:23.704607 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:37:23.729279 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
I20240502 14:37:23.753130 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:23.753893 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:23.754154 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:23.754444 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:23.755537 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:23.757932 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:37:23.778541 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:23.784478 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:37:23.902098 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:23.969250 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
I20240502 14:37:23.980659 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:23.984768 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:23.988097 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:23.992415 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:37:23.999233 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
I20240502 14:37:24.017997 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:24.087586 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 14:37:24.096024 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:37:24.232823 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:37:24.251225 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:24.252871 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:24.264534 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:24.265004 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:24.287653 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:37:24.308234 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:37:24.324527 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:24.355834 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:37:24.436816 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
I20240502 14:37:24.448566 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:24.460705 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:24.461654 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:24.467259 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:37:24.468565 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:24.479846 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:24.521610 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 14:37:24.569500 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:37:24.668506 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:37:24.677836 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:37:24.684257 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:24.684500 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:37:24.697803 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
I20240502 14:37:24.711076 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:24.714622 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:24.714823 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:24.786337 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:24.935699 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:24.938792 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:24.954279 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:24.957111 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:37:24.962275 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:24.968968 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:25.081981 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:37:25.155606 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:37:25.171234 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:25.174997 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:25.187290 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:25.187335 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:25.208966 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:25.261413 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:25.288728 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:37:25.364440 18768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 100 similar messages]
I20240502 14:37:25.387329 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:25.389091 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:25.410210 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:25.417577 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:25.429958 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:25.554908 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:25.571831 18900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240502 14:37:25.628612 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:25.645280 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:37:25.672245 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:25.679725 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:25.681774 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:25.713634 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:25.806291 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:37:25.812090 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:37:25.928443 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:25.932006 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:25.942427 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:25.944423 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:25.947202 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:26.010255 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:37:26.027246 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:26.163758 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:37:26.166095 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:26.167028 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:26.170217 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:26.173678 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:26.187570 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:26.205524 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:26.223001 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:37:26.306224 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20240502 14:37:26.334873 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:37:26.398910 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:37:26.400079 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:37:26.408308 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:26.414783 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:26.417723 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:26.420857 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:26.433493 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:26.529371 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20240502 14:37:26.601717 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:37:26.615167 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:26.618445 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:26.630882 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:26.640384 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:26.676759 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:26.738857 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20240502 14:37:26.774444 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:26.778910 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:37:26.870343 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:26.877493 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:26.878144 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:26.887290 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:26.901852 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:26.902844 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:37:26.908538 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:37:26.930706 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:37:26.989722 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20240502 14:37:27.034268 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:37:27.039042 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:37:27.106540 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:27.110301 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:27.111068 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:27.114024 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:27.115217 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:27.209511 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20240502 14:37:27.253520 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:37:27.285905 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:27.326272 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:27.339439 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:27.344027 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:27.344516 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:27.351408 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:27.430519 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:37:27.472704 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:37:27.529865 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:27.532359 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:27.548342 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:27.552171 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:27.553170 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:27.566922 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:27.701951 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:37:27.766194 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:27.771301 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:27.776084 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:27.790956 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:27.836004 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:27.863040 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:27.998297 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:28.003405 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:28.014081 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:28.017972 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:28.019421 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:28.130940 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 27 similar messages]
W20240502 14:37:28.132030 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:37:28.231859 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:28.233276 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:28.264860 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:28.271648 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:28.291915 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:37:28.298767 20561 consensus_peers.cc:485] T 51ef0be93c914e258635e13f3cca79aa P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:37:28.303469 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:28.399338 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:28.457386 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:28.468235 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:28.482270 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:28.484371 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:28.485023 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:28.586067 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:37:28.601387 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 14:37:28.648021 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:37:28.693297 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:28.704941 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:28.731176 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:28.732664 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:37:28.734117 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:28.743657 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:28.786188 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:37:28.834195 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:28.848546 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20240502 14:37:28.880196 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:37:28.957453 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:28.960065 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:28.963573 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:28.974737 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:28.997776 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:37:29.110602 18636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 14:37:29.113276 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:29.140399 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20240502 14:37:29.167907 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:37:29.194005 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:29.202836 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:37:29.202836 20561 consensus_peers.cc:485] T 522014370f0a4cc3bd31aa00d6a05c5e P a834edffbcb2469aae598b069aab0681 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:37:29.210223 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:29.227173 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:29.250892 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:29.271766 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:37:29.277396 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
W20240502 14:37:29.366123 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:37:29.395046 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:29.415797 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:37:29.424503 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:29.434194 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:29.436527 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:29.447342 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:29.463380 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:29.477854 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20240502 14:37:29.553372 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:37:29.570988 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:29.623940 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
I20240502 14:37:29.640039 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:29.645501 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:29.665932 29244 raft_consensus.cc:998] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b: Attempting to remove follower 7ab6d395b30d4afb9821b61c9454430b from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7ab6d395b30d4afb9821b61c9454430b for more than 300 seconds (300.209s)
I20240502 14:37:29.667451 29244 consensus_queue.cc:237] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:37:29.669368 18962 raft_consensus.cc:1261] T 96662a3d9e1b4f03b05464ddd06b93d6 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:37:29.670086 29244 consensus_queue.cc:1035] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 14:37:29.686530 20561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 14:37:29.693950 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:29.727083 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:29.741744 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:37:29.762284 29363 raft_consensus.cc:2932] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
I20240502 14:37:29.766288 18965 raft_consensus.cc:2932] T 96662a3d9e1b4f03b05464ddd06b93d6 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
I20240502 14:37:29.774772 18551 catalog_manager.cc:5484] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b reported cstate change: config changed from index 7 to 8, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) evicted. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
I20240502 14:37:29.789659 29372 raft_consensus.cc:998] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06: Attempting to remove follower 7ab6d395b30d4afb9821b61c9454430b from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7ab6d395b30d4afb9821b61c9454430b for more than 300 seconds (300.021s)
I20240502 14:37:29.793569 29372 consensus_queue.cc:237] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:37:29.800889 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:29.805862 19095 raft_consensus.cc:1261] T 246a8425400f4890a5264c4ae8a627c5 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:37:29.806428 29374 consensus_queue.cc:1035] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:37:29.819169 19093 raft_consensus.cc:2932] T 246a8425400f4890a5264c4ae8a627c5 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
I20240502 14:37:29.825373 29374 raft_consensus.cc:2932] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
I20240502 14:37:29.855922 18552 catalog_manager.cc:5484] T 246a8425400f4890a5264c4ae8a627c5 P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: config changed from index 7 to 8, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) evicted. New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
W20240502 14:37:29.875437 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:37:29.890252 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:29.891297 18546 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111) [suppressed 18 similar messages]
I20240502 14:37:29.900179 18965 consensus_queue.cc:237] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
W20240502 14:37:29.902463 18900 consensus_peers.cc:485] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 246a8425400f4890a5264c4ae8a627c5. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:37:29.902704 19095 raft_consensus.cc:1261] T 246a8425400f4890a5264c4ae8a627c5 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Refusing update from remote peer c902805060df4df9b5599d0e94b2dc06: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:37:29.903398 29377 consensus_queue.cc:1035] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:37:29.903627 18699 consensus_queue.cc:237] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
W20240502 14:37:29.903817 18546 catalog_manager.cc:4573] TS 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 96662a3d9e1b4f03b05464ddd06b93d6: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:37:29.905905 18965 raft_consensus.cc:1261] T 96662a3d9e1b4f03b05464ddd06b93d6 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:37:29.906204 29377 raft_consensus.cc:2932] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:37:29.909446 29244 consensus_queue.cc:1035] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:37:29.912387 29363 raft_consensus.cc:2932] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:37:29.915647 18965 raft_consensus.cc:2932] T 96662a3d9e1b4f03b05464ddd06b93d6 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:37:29.917097 18546 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 246a8425400f4890a5264c4ae8a627c5 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
W20240502 14:37:29.918277 18637 consensus_peers.cc:485] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b -> Peer 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463): Couldn't send request to peer 007b2e79dc424a6f80141180f072f2d8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:37:29.918792 19095 raft_consensus.cc:2932] T 246a8425400f4890a5264c4ae8a627c5 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:37:29.919409 18552 catalog_manager.cc:5484] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 reported cstate change: config changed from index 8 to 9, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New cstate: current_term: 1 leader_uuid: "c902805060df4df9b5599d0e94b2dc06" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:37:29.924422 18544 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 96662a3d9e1b4f03b05464ddd06b93d6 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:37:29.927853 18553 catalog_manager.cc:5484] T 96662a3d9e1b4f03b05464ddd06b93d6 P c902805060df4df9b5599d0e94b2dc06 reported cstate change: config changed from index 8 to 9, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) added. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:37:29.955940 29186 raft_consensus.cc:998] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4: Attempting to remove follower 7ab6d395b30d4afb9821b61c9454430b from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7ab6d395b30d4afb9821b61c9454430b for more than 300 seconds (300.455s)
I20240502 14:37:29.956771 29186 consensus_queue.cc:237] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
I20240502 14:37:29.961831 18965 raft_consensus.cc:1261] T 67827c0262724897a05bf80df579bbf3 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:37:29.962450 29399 consensus_queue.cc:1035] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:37:29.965137 29186 raft_consensus.cc:2932] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
I20240502 14:37:29.965729 18965 raft_consensus.cc:2932] T 67827c0262724897a05bf80df579bbf3 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
I20240502 14:37:29.970731 18552 catalog_manager.cc:5484] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index 7 to 8, VOTER 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71) evicted. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
I20240502 14:37:29.982488 18832 consensus_queue.cc:237] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
W20240502 14:37:29.984869 18769 consensus_peers.cc:485] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68:41463): Couldn't send request to peer 007b2e79dc424a6f80141180f072f2d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67827c0262724897a05bf80df579bbf3. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:37:29.985515 18962 raft_consensus.cc:1261] T 67827c0262724897a05bf80df579bbf3 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer fec04c50dc5b49878ffa3c3873df32e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:37:29.986471 29399 consensus_queue.cc:1035] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:37:29.988811 29186 raft_consensus.cc:2932] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:37:29.990787 18962 raft_consensus.cc:2932] T 67827c0262724897a05bf80df579bbf3 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:37:29.992996 18543 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 67827c0262724897a05bf80df579bbf3 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:37:29.995136 18553 catalog_manager.cc:5484] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index 8 to 9, VOTER 007b2e79dc424a6f80141180f072f2d8 (127.5.107.68) added. New cstate: current_term: 1 leader_uuid: "fec04c50dc5b49878ffa3c3873df32e4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } }
I20240502 14:37:30.043136 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:30.053648 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:30.064002 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:30.072583 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:30.081212 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:30.167093 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:30.228016 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:37:30.279691 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:30.286607 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:30.290161 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:37:30.307277 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:30.313604 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:30.397616 18768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111) [suppressed 101 similar messages]
I20240502 14:37:30.425419 29486 ts_tablet_manager.cc:902] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8: Initiating tablet copy from peer 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:37:30.430534 29486 tablet_copy_client.cc:308] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.65:36749
I20240502 14:37:30.431200 18721 tablet_copy_service.cc:140] P 2db304b538e14857b7b1cc272d0db94b: Received BeginTabletCopySession request for tablet 96662a3d9e1b4f03b05464ddd06b93d6 from peer 007b2e79dc424a6f80141180f072f2d8 ({username='slave'} at 127.5.107.68:35397)
I20240502 14:37:30.431380 18721 tablet_copy_service.cc:161] P 2db304b538e14857b7b1cc272d0db94b: Beginning new tablet copy session on tablet 96662a3d9e1b4f03b05464ddd06b93d6 from peer 007b2e79dc424a6f80141180f072f2d8 at {username='slave'} at 127.5.107.68:35397: session id = 007b2e79dc424a6f80141180f072f2d8-96662a3d9e1b4f03b05464ddd06b93d6
I20240502 14:37:30.435706 18721 tablet_copy_source_session.cc:205] T 96662a3d9e1b4f03b05464ddd06b93d6 P 2db304b538e14857b7b1cc272d0db94b: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:37:30.437327 29486 ts_tablet_manager.cc:1875] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8: Deleting tablet data with delete state TABLET_DATA_COPYING
W20240502 14:37:30.438648 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:30.448201 29486 ts_tablet_manager.cc:1888] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 14:37:30.448442 29486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96662a3d9e1b4f03b05464ddd06b93d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:37:30.451330 29486 tablet_copy_client.cc:783] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8: tablet copy: Starting download of 5 data blocks...
I20240502 14:37:30.458741 29486 tablet_copy_client.cc:647] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8: tablet copy: Starting download of 1 WAL segments...
I20240502 14:37:30.461745 29486 tablet_copy_client.cc:523] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:37:30.465301 29486 tablet_bootstrap.cc:492] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8: Bootstrap starting.
I20240502 14:37:30.511354 29486 tablet_bootstrap.cc:492] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:37:30.512154 29486 tablet_bootstrap.cc:492] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8: Bootstrap complete.
I20240502 14:37:30.515331 29486 ts_tablet_manager.cc:1367] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8: Time spent bootstrapping tablet: real 0.050s user 0.018s sys 0.005s
I20240502 14:37:30.522504 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:30.523239 29486 raft_consensus.cc:348] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:37:30.523646 29486 raft_consensus.cc:729] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Initialized, Role: FOLLOWER
I20240502 14:37:30.524093 29486 consensus_queue.cc:260] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:37:30.531505 29502 ts_tablet_manager.cc:902] T 246a8425400f4890a5264c4ae8a627c5 P a834edffbcb2469aae598b069aab0681: Initiating tablet copy from peer c902805060df4df9b5599d0e94b2dc06 (127.5.107.67:45409)
F20240502 14:37:30.532130 29502 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-097eb705c9e24740a5be61bf3354f3d8.queue_time_us, tablet-download-pool-097eb705c9e24740a5be61bf3354f3d8.queue_time_us) (tablet-download-pool-097eb705c9e24740a5be61bf3354f3d8.run_cpu_time_us, tablet-download-pool-097eb705c9e24740a5be61bf3354f3d8.run_cpu_time_us) (tablet-download-pool-097eb705c9e24740a5be61bf3354f3d8.run_wall_time_us, tablet-download-pool-097eb705c9e24740a5be61bf3354f3d8.run_wall_time_us) (tablet-download-pool-21af917743444f2db932636a78f33512.queue_time_us, tablet-download-pool-21af917743444f2db932636a78f33512.queue_time_us) (tablet-download-pool-21af917743444f2db932636a78f33512.run_cpu_time_us, tablet-download-pool-21af917743444f2db932636a78f33512.run_cpu_time_us) (tablet-download-pool-21af917743444f2db932636a78f33512.run_wall_time_us, tablet-download-pool-21af917743444f2db932636a78f33512.run_wall_time_us) (tablet-download-pool-246a8425400f4890a5264c4ae8a627c5.queue_time_us, tablet-download-pool-246a8425400f4890a5264c4ae8a627c5.queue_time_us) (tablet-download-pool-2c2afaffde96462eacd0d32bbdec23b1.queue_time_us, tablet-download-pool-2c2afaffde96462eacd0d32bbdec23b1.queue_time_us) (tablet-download-pool-2c2afaffde96462eacd0d32bbdec23b1.run_cpu_time_us, tablet-download-pool-2c2afaffde96462eacd0d32bbdec23b1.run_cpu_time_us) (tablet-download-pool-2c2afaffde96462eacd0d32bbdec23b1.run_wall_time_us, tablet-download-pool-2c2afaffde96462eacd0d32bbdec23b1.run_wall_time_us) (tablet-download-pool-51ef0be93c914e258635e13f3cca79aa.queue_time_us, tablet-download-pool-51ef0be93c914e258635e13f3cca79aa.queue_time_us) (tablet-download-pool-51ef0be93c914e258635e13f3cca79aa.run_cpu_time_us, tablet-download-pool-51ef0be93c914e258635e13f3cca79aa.run_cpu_time_us) (tablet-download-pool-51ef0be93c914e258635e13f3cca79aa.run_wall_time_us, tablet-download-pool-51ef0be93c914e258635e13f3cca79aa.run_wall_time_us) (tablet-download-pool-522014370f0a4cc3bd31aa00d6a05c5e.queue_time_us, tablet-download-pool-522014370f0a4cc3bd31aa00d6a05c5e.queue_time_us) (tablet-download-pool-522014370f0a4cc3bd31aa00d6a05c5e.run_cpu_time_us, tablet-download-pool-522014370f0a4cc3bd31aa00d6a05c5e.run_cpu_time_us) (tablet-download-pool-522014370f0a4cc3bd31aa00d6a05c5e.run_wall_time_us, tablet-download-pool-522014370f0a4cc3bd31aa00d6a05c5e.run_wall_time_us) (tablet-download-pool-6a992d12aee7416096d1353533397386.queue_time_us, tablet-download-pool-6a992d12aee7416096d1353533397386.queue_time_us) (tablet-download-pool-6a992d12aee7416096d1353533397386.run_cpu_time_us, tablet-download-pool-6a992d12aee7416096d1353533397386.run_cpu_time_us) (tablet-download-pool-6a992d12aee7416096d1353533397386.run_wall_time_us, tablet-download-pool-6a992d12aee7416096d1353533397386.run_wall_time_us) (tablet-download-pool-8937c809610542d88fdde3daf002c890.queue_time_us, tablet-download-pool-8937c809610542d88fdde3daf002c890.queue_time_us) (tablet-download-pool-8937c809610542d88fdde3daf002c890.run_cpu_time_us, tablet-download-pool-8937c809610542d88fdde3daf002c890.run_cpu_time_us) (tablet-download-pool-8937c809610542d88fdde3daf002c890.run_wall_time_us, tablet-download-pool-8937c809610542d88fdde3daf002c890.run_wall_time_us) (tablet-download-pool-8a2bed29fcd64f7bbb04286404ef09de.queue_time_us, tablet-download-pool-8a2bed29fcd64f7bbb04286404ef09de.queue_time_us) (tablet-download-pool-8a2bed29fcd64f7bbb04286404ef09de.run_cpu_time_us, tablet-download-pool-8a2bed29fcd64f7bbb04286404ef09de.run_cpu_time_us) (tablet-download-pool-8a2bed29fcd64f7bbb04286404ef09de.run_wall_time_us, tablet-download-pool-8a2bed29fcd64f7bbb04286404ef09de.run_wall_time_us) (tablet-download-pool-9580f3fdf7084cc0923b1b16bfedd450.queue_time_us, tablet-download-pool-9580f3fdf7084cc0923b1b16bfedd450.queue_time_us) (tablet-download-pool-9580f3fdf7084cc0923b1b16bfedd450.run_cpu_time_us, tablet-download-pool-9580f3fdf7084cc0923b1b16bfedd450.run_cpu_time_us) (tablet-download-pool-9580f3fdf7084cc0923b1b16bfedd450.run_wall_time_us, tablet-download-pool-9580f3fdf7084cc0923b1b16bfedd450.run_wall_time_us) (tablet-download-pool-b13bcd6e693f4c7687a1aabdafbb83c6.queue_time_us, tablet-download-pool-b13bcd6e693f4c7687a1aabdafbb83c6.queue_time_us) (tablet-download-pool-b13bcd6e693f4c7687a1aabdafbb83c6.run_cpu_time_us, tablet-download-pool-b13bcd6e693f4c7687a1aabdafbb83c6.run_cpu_time_us) (tablet-download-pool-b13bcd6e693f4c7687a1aabdafbb83c6.run_wall_time_us, tablet-download-pool-b13bcd6e693f4c7687a1aabdafbb83c6.run_wall_time_us) (tablet-download-pool-d28db16c9f7e405b8866f0e6c548e246.queue_time_us, tablet-download-pool-d28db16c9f7e405b8866f0e6c548e246.queue_time_us) (tablet-download-pool-d28db16c9f7e405b8866f0e6c548e246.run_cpu_time_us, tablet-download-pool-d28db16c9f7e405b8866f0e6c548e246.run_cpu_time_us) (tablet-download-pool-d28db16c9f7e405b8866f0e6c548e246.run_wall_time_us, tablet-download-pool-d28db16c9f7e405b8866f0e6c548e246.run_wall_time_us) (tablet-download-pool-db56136f2dbb429098d4d78b15a3a439.queue_time_us, tablet-download-pool-db56136f2dbb429098d4d78b15a3a439.queue_time_us) (tablet-download-pool-db56136f2dbb429098d4d78b15a3a439.run_cpu_time_us, tablet-download-pool-db56136f2dbb429098d4d78b15a3a439.run_cpu_time_us) (tablet-download-pool-db56136f2dbb429098d4d78b15a3a439.run_wall_time_us, tablet-download-pool-db56136f2dbb429098d4d78b15a3a439.run_wall_time_us) (tablet-download-pool-df5f14206a1d46e4866f34fa234f945b.queue_time_us, tablet-download-pool-df5f14206a1d46e4866f34fa234f945b.queue_time_us) (tablet-download-pool-df5f14206a1d46e4866f34fa234f945b.run_cpu_time_us, tablet-download-pool-df5f14206a1d46e4866f34fa234f945b.run_cpu_time_us) (tablet-download-pool-df5f14206a1d46e4866f34fa234f945b.run_wall_time_us, tablet-download-pool-df5f14206a1d46e4866f34fa234f945b.run_wall_time_us) (tablet-download-pool-ecb8a9b04ba34f779e2227e77e6f764c.queue_time_us, tablet-download-pool-ecb8a9b04ba34f779e2227e77e6f764c.queue_time_us) (tablet-download-pool-ecb8a9b04ba34f779e2227e77e6f764c.run_cpu_time_us, tablet-download-pool-ecb8a9b04ba34f779e2227e77e6f764c.run_cpu_time_us) (tablet-download-pool-ecb8a9b04ba34f779e2227e77e6f764c.run_wall_time_us, tablet-download-pool-ecb8a9b04ba34f779e2227e77e6f764c.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
I20240502 14:37:30.532958 29486 ts_tablet_manager.cc:1398] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
*** Check failure stack trace: ***
@ 0x7f13fdb04dcd google::LogMessage::Fail() at ??:0
I20240502 14:37:30.536636 18721 tablet_copy_service.cc:342] P 2db304b538e14857b7b1cc272d0db94b: Request end of tablet copy session 007b2e79dc424a6f80141180f072f2d8-96662a3d9e1b4f03b05464ddd06b93d6 received from {username='slave'} at 127.5.107.68:35397
I20240502 14:37:30.536762 18721 tablet_copy_service.cc:433] P 2db304b538e14857b7b1cc272d0db94b: ending tablet copy session 007b2e79dc424a6f80141180f072f2d8-96662a3d9e1b4f03b05464ddd06b93d6 on tablet 96662a3d9e1b4f03b05464ddd06b93d6 with peer 007b2e79dc424a6f80141180f072f2d8
@ 0x7f13fdb08b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f13fdb047cc google::LogMessage::Flush() at ??:0
@ 0x7f13fdb05f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 14:37:30.555802 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:30.556381 20607 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
@ 0x7f1400845742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f1400811f8b kudu::ThreadPool::ThreadPool() at ??:0
I20240502 14:37:30.573114 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
@ 0x7f140080bcd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240502 14:37:30.578795 29486 ts_tablet_manager.cc:902] T 67827c0262724897a05bf80df579bbf3 P 007b2e79dc424a6f80141180f072f2d8: Initiating tablet copy from peer fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009)
@ 0x7f140fc5e8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f140fc76daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f140fda1480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f140fdd9a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f140fdbc981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f140fdbc70e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240502 14:37:30.643178 29486 tablet_copy_client.cc:308] T 67827c0262724897a05bf80df579bbf3 P 007b2e79dc424a6f80141180f072f2d8: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.66:33009
I20240502 14:37:30.644395 18854 tablet_copy_service.cc:140] P fec04c50dc5b49878ffa3c3873df32e4: Received BeginTabletCopySession request for tablet 67827c0262724897a05bf80df579bbf3 from peer 007b2e79dc424a6f80141180f072f2d8 ({username='slave'} at 127.5.107.68:50349)
I20240502 14:37:30.644529 18854 tablet_copy_service.cc:161] P fec04c50dc5b49878ffa3c3873df32e4: Beginning new tablet copy session on tablet 67827c0262724897a05bf80df579bbf3 from peer 007b2e79dc424a6f80141180f072f2d8 at {username='slave'} at 127.5.107.68:50349: session id = 007b2e79dc424a6f80141180f072f2d8-67827c0262724897a05bf80df579bbf3
@ 0x7f1400815b75 kudu::ThreadPool::DispatchThread() at ??:0
W20240502 14:37:30.645804 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:37:30.648540 18854 tablet_copy_source_session.cc:205] T 67827c0262724897a05bf80df579bbf3 P fec04c50dc5b49878ffa3c3873df32e4: Tablet Copy: opened 5 blocks and 1 log segments
@ 0x7f140081919c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
I20240502 14:37:30.650139 29486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67827c0262724897a05bf80df579bbf3. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f1400819019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f14007f401e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1403b796db start_thread at ??:0
@ 0x7f13fc9b771f clone at ??:0
I20240502 14:37:30.666862 29486 tablet_copy_client.cc:783] T 67827c0262724897a05bf80df579bbf3 P 007b2e79dc424a6f80141180f072f2d8: tablet copy: Starting download of 5 data blocks...
W20240502 14:37:30.674714 18636 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:37:30.683591 29486 tablet_copy_client.cc:647] T 67827c0262724897a05bf80df579bbf3 P 007b2e79dc424a6f80141180f072f2d8: tablet copy: Starting download of 1 WAL segments...
I20240502 14:37:30.685176 29486 tablet_copy_client.cc:523] T 67827c0262724897a05bf80df579bbf3 P 007b2e79dc424a6f80141180f072f2d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:37:30.689059 29486 tablet_bootstrap.cc:492] T 67827c0262724897a05bf80df579bbf3 P 007b2e79dc424a6f80141180f072f2d8: Bootstrap starting.
I20240502 14:37:30.711434 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:30.733500 19143 maintenance_manager.cc:419] P 007b2e79dc424a6f80141180f072f2d8: Scheduling UndoDeltaBlockGCOp(96662a3d9e1b4f03b05464ddd06b93d6): 4586 bytes on disk
I20240502 14:37:30.734066 19033 maintenance_manager.cc:643] P 007b2e79dc424a6f80141180f072f2d8: UndoDeltaBlockGCOp(96662a3d9e1b4f03b05464ddd06b93d6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240502 14:37:30.769237 29486 tablet_bootstrap.cc:492] T 67827c0262724897a05bf80df579bbf3 P 007b2e79dc424a6f80141180f072f2d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:37:30.769708 29486 tablet_bootstrap.cc:492] T 67827c0262724897a05bf80df579bbf3 P 007b2e79dc424a6f80141180f072f2d8: Bootstrap complete.
I20240502 14:37:30.770025 29486 ts_tablet_manager.cc:1367] T 67827c0262724897a05bf80df579bbf3 P 007b2e79dc424a6f80141180f072f2d8: Time spent bootstrapping tablet: real 0.081s user 0.022s sys 0.002s
I20240502 14:37:30.770843 29486 raft_consensus.cc:348] T 67827c0262724897a05bf80df579bbf3 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:37:30.771092 29486 raft_consensus.cc:729] T 67827c0262724897a05bf80df579bbf3 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007b2e79dc424a6f80141180f072f2d8, State: Initialized, Role: FOLLOWER
I20240502 14:37:30.771345 29486 consensus_queue.cc:260] T 67827c0262724897a05bf80df579bbf3 P 007b2e79dc424a6f80141180f072f2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } }
I20240502 14:37:30.771883 29486 ts_tablet_manager.cc:1398] T 67827c0262724897a05bf80df579bbf3 P 007b2e79dc424a6f80141180f072f2d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:37:30.773695 18854 tablet_copy_service.cc:342] P fec04c50dc5b49878ffa3c3873df32e4: Request end of tablet copy session 007b2e79dc424a6f80141180f072f2d8-67827c0262724897a05bf80df579bbf3 received from {username='slave'} at 127.5.107.68:50349
I20240502 14:37:30.773799 18854 tablet_copy_service.cc:433] P fec04c50dc5b49878ffa3c3873df32e4: ending tablet copy session 007b2e79dc424a6f80141180f072f2d8-67827c0262724897a05bf80df579bbf3 on tablet 67827c0262724897a05bf80df579bbf3 with peer 007b2e79dc424a6f80141180f072f2d8
W20240502 14:37:30.852118 18898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 14:37:30.901118 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:30.905372 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:30.925803 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:30.930893 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:30.930933 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:30.974959 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:37:30.986780 19143 maintenance_manager.cc:419] P 007b2e79dc424a6f80141180f072f2d8: Scheduling UndoDeltaBlockGCOp(67827c0262724897a05bf80df579bbf3): 4840 bytes on disk
I20240502 14:37:30.987526 19033 maintenance_manager.cc:643] P 007b2e79dc424a6f80141180f072f2d8: UndoDeltaBlockGCOp(67827c0262724897a05bf80df579bbf3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4}
W20240502 14:37:31.008100 18546 catalog_manager.cc:4573] TS 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 96662a3d9e1b4f03b05464ddd06b93d6: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:37:31.008352 19095 raft_consensus.cc:1203] T 67827c0262724897a05bf80df579bbf3 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 14:37:31.031325 19095 raft_consensus.cc:1203] T 96662a3d9e1b4f03b05464ddd06b93d6 P 007b2e79dc424a6f80141180f072f2d8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
W20240502 14:37:31.034148 18900 connection.cc:535] client connection to 127.5.107.69:41119 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:37:31.034583 19032 connection.cc:535] client connection to 127.5.107.69:41119 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:37:31.036055 18637 connection.cc:535] client connection to 127.5.107.69:41119 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:37:31.036315 18900 consensus_peers.cc:462] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111)
W20240502 14:37:31.036427 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:37:31.036518 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:37:31.036584 18900 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:37:31.038209 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:37:31.038445 18769 connection.cc:535] client connection to 127.5.107.69:41119 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:37:31.039836 18637 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:37:31.039944 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:37:31.041363 18769 consensus_peers.cc:485] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:37:31.041466 18769 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:37:31.043795 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:37:31.083135 18900 consensus_peers.cc:462] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111)
W20240502 14:37:31.107903 18637 consensus_peers.cc:485] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:37:31.125981 19032 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:37:31.148471 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:31.162295 18636 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:37:31.192477 19032 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:37:31.280553 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:37:31.357398 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 14:37:31.395124 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1201: this message will repeat every 5th retry.
I20240502 14:37:31.404510 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:31.418740 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:31.435571 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:31.440016 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:31.446730 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:31.462951 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:31.469928 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:31.555079 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:31.563822 18900 consensus_peers.cc:462] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111)
I20240502 14:37:31.584867 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:31.601559 19031 consensus_peers.cc:462] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:31.652493 18636 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:37:31.715265 18637 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 14:37:31.719446 18769 consensus_peers.cc:485] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20240502 14:37:31.807647 18768 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:37:31.846742 18769 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
I20240502 14:37:31.859236 29650 raft_consensus.cc:482] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a834edffbcb2469aae598b069aab0681)
I20240502 14:37:31.859414 29650 raft_consensus.cc:504] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:37:31.860086 29650 leader_election.cc:288] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871), a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119)
W20240502 14:37:31.862473 18636 leader_election.cc:334] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
W20240502 14:37:31.871098 18637 leader_election.cc:334] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111)
I20240502 14:37:31.871215 18637 leader_election.cc:302] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2db304b538e14857b7b1cc272d0db94b; no voters: a834edffbcb2469aae598b069aab0681, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:37:31.871460 29650 raft_consensus.cc:2726] T 522014370f0a4cc3bd31aa00d6a05c5e P 2db304b538e14857b7b1cc272d0db94b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 14:37:31.881213 18769 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
W20240502 14:37:31.904858 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:31.933552 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:31.946651 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:31.951735 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:31.957907 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:31.965975 18900 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): Couldn't send request to peer 7ab6d395b30d4afb9821b61c9454430b. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20240502 14:37:31.974643 18768 consensus_peers.cc:485] T 332e4b23751b422ab7bd17b80c93701d P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:37:31.974848 18900 consensus_peers.cc:462] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111)
I20240502 14:37:32.069356 28424 raft_consensus.cc:998] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8: Attempting to remove follower ff6acba046404ef0a6f41ad5e09d852f from the Raft config. Reason: Leader has been unable to successfully communicate with peer ff6acba046404ef0a6f41ad5e09d852f for more than 300 seconds (300.247s)
I20240502 14:37:32.070035 28424 consensus_queue.cc:237] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:37:32.072906 18832 raft_consensus.cc:1261] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:37:32.073591 29656 consensus_queue.cc:1035] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 14:37:32.075604 18546 catalog_manager.cc:4573] TS 7ab6d395b30d4afb9821b61c9454430b (127.5.107.71:39195): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 96662a3d9e1b4f03b05464ddd06b93d6: Network error: Client connection negotiation failed: client connection to 127.5.107.71:39195: connect: Connection refused (error 111)
I20240502 14:37:32.083050 28424 raft_consensus.cc:2932] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } }
I20240502 14:37:32.089107 18832 raft_consensus.cc:2932] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } }
I20240502 14:37:32.096078 18551 catalog_manager.cc:5484] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4 reported cstate change: config changed from index 9 to 10, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) evicted. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } }
I20240502 14:37:32.108536 29372 raft_consensus.cc:482] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a834edffbcb2469aae598b069aab0681)
I20240502 14:37:32.108743 29372 raft_consensus.cc:504] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "ff6acba046404ef0a6f41ad5e09d852f" member_type: VOTER last_known_addr { host: "127.5.107.72" port: 35871 } }
I20240502 14:37:32.109745 29372 leader_election.cc:288] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119), ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871)
W20240502 14:37:32.111582 18900 leader_election.cc:334] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111)
I20240502 14:37:32.111944 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:37:32.112063 18899 leader_election.cc:334] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:37:32.112171 18899 leader_election.cc:302] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c902805060df4df9b5599d0e94b2dc06; no voters: a834edffbcb2469aae598b069aab0681, ff6acba046404ef0a6f41ad5e09d852f
I20240502 14:37:32.112751 29372 raft_consensus.cc:2726] T 51ef0be93c914e258635e13f3cca79aa P c902805060df4df9b5599d0e94b2dc06 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:37:32.118249 19095 consensus_queue.cc:237] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } }
I20240502 14:37:32.120657 18832 raft_consensus.cc:1261] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Refusing update from remote peer 007b2e79dc424a6f80141180f072f2d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:37:32.121580 28424 consensus_queue.cc:1035] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 14:37:32.121898 19032 consensus_peers.cc:485] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:37:32.125990 28424 raft_consensus.cc:2932] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:37:32.137095 18546 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet e5fb3b049ba94af998791640bd0e31c5 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:37:32.148391 18832 raft_consensus.cc:2932] T e5fb3b049ba94af998791640bd0e31c5 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:37:32.149482 18551 catalog_manager.cc:5484] T e5fb3b049ba94af998791640bd0e31c5 P 007b2e79dc424a6f80141180f072f2d8 reported cstate change: config changed from index 10 to 11, VOTER a834edffbcb2469aae598b069aab0681 (127.5.107.69) added. New cstate: current_term: 1 leader_uuid: "007b2e79dc424a6f80141180f072f2d8" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "007b2e79dc424a6f80141180f072f2d8" member_type: VOTER last_known_addr { host: "127.5.107.68" port: 41463 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } }
I20240502 14:37:32.151976 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:32.153378 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:32.158367 18768 consensus_peers.cc:485] T 8b8d81c275144f898dcc69a6d98d11e0 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 14:37:32.175246 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:32.207844 29650 raft_consensus.cc:998] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b: Attempting to remove follower ff6acba046404ef0a6f41ad5e09d852f from the Raft config. Reason: Leader has been unable to successfully communicate with peer ff6acba046404ef0a6f41ad5e09d852f for more than 300 seconds (300.236s)
I20240502 14:37:32.208611 29650 consensus_queue.cc:237] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
I20240502 14:37:32.218375 18832 raft_consensus.cc:1261] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:37:32.219236 29749 consensus_queue.cc:1035] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 14:37:32.222019 18637 consensus_peers.cc:485] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:37:32.225849 29749 raft_consensus.cc:2932] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } }
I20240502 14:37:32.226583 18832 raft_consensus.cc:2932] T db56136f2dbb429098d4d78b15a3a439 P fec04c50dc5b49878ffa3c3873df32e4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } }
I20240502 14:37:32.232168 18551 catalog_manager.cc:5484] T db56136f2dbb429098d4d78b15a3a439 P 2db304b538e14857b7b1cc272d0db94b reported cstate change: config changed from index 8 to 9, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) evicted. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "a834edffbcb2469aae598b069aab0681" member_type: VOTER last_known_addr { host: "127.5.107.69" port: 41119 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } }
W20240502 14:37:32.265278 19031 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:37:32.357656 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:32.364051 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:32.378928 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:32.379029 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:32.393815 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:37:32.516569 18900 consensus_peers.cc:462] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111)
I20240502 14:37:32.593170 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:32.597074 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:32.621583 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:32.628803 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:32.764444 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:32.834551 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:37:32.855269 18768 consensus_peers.cc:485] T 0d8667d91fd64809b635362c4a8fa33f P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 14:37:32.855502 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:32.866341 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:32.870098 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:33.038915 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:33.043884 18900 consensus_peers.cc:462] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111)
I20240502 14:37:33.049855 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:33.058189 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:33.068274 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:33.134660 19032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111) [suppressed 44 similar messages]
I20240502 14:37:33.254581 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:33.262377 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:33.271642 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:33.288697 29748 raft_consensus.cc:998] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b: Attempting to remove follower ff6acba046404ef0a6f41ad5e09d852f from the Raft config. Reason: Leader has been unable to successfully communicate with peer ff6acba046404ef0a6f41ad5e09d852f for more than 300 seconds (300.279s)
I20240502 14:37:33.289790 29748 consensus_queue.cc:237] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } }
I20240502 14:37:33.298354 18965 raft_consensus.cc:1261] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [term 3 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240502 14:37:33.298919 29953 consensus_queue.cc:1035] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240502 14:37:33.304553 18769 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:37:33.305390 18545 catalog_manager.cc:4573] TS ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet e5fb3b049ba94af998791640bd0e31c5: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111)
I20240502 14:37:33.314424 29953 raft_consensus.cc:2932] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
W20240502 14:37:33.318491 18769 consensus_peers.cc:485] T 8a2bed29fcd64f7bbb04286404ef09de P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:37:33.318956 18965 raft_consensus.cc:2932] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:37:33.325791 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:33.330694 18551 catalog_manager.cc:5484] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 reported cstate change: config changed from index 10 to 12, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) evicted. New cstate: current_term: 3 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } }
I20240502 14:37:33.338337 18701 consensus_queue.cc:237] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
W20240502 14:37:33.340723 18634 consensus_peers.cc:485] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b -> Peer fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009): Couldn't send request to peer fec04c50dc5b49878ffa3c3873df32e4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b15d11209e45473e863766e5cd6e822f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:37:33.340704 18965 raft_consensus.cc:1261] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [term 3 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 14:37:33.341301 29954 consensus_queue.cc:1035] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 14:37:33.344041 29954 raft_consensus.cc:2932] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } }
I20240502 14:37:33.348646 18965 raft_consensus.cc:2932] T b15d11209e45473e863766e5cd6e822f P c902805060df4df9b5599d0e94b2dc06 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } }
I20240502 14:37:33.349280 18544 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet b15d11209e45473e863766e5cd6e822f with cas_config_opid_index 12: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:37:33.351696 18556 catalog_manager.cc:5484] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b reported cstate change: config changed from index 12 to 13, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) added. New cstate: current_term: 3 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } }
W20240502 14:37:33.378926 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
W20240502 14:37:33.380844 18900 consensus_peers.cc:485] T b13bcd6e693f4c7687a1aabdafbb83c6 P c902805060df4df9b5599d0e94b2dc06 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:37:33.407837 18898 consensus_peers.cc:485] T cf2a6426ae6c438f942d39530da6e3e3 P c902805060df4df9b5599d0e94b2dc06 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Couldn't send request to peer a14c220ad2cc4bf9bd7dad65f4803eb0. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111). This is attempt 1206: this message will repeat every 5th retry.
W20240502 14:37:33.445235 18900 consensus_peers.cc:485] T 21af917743444f2db932636a78f33512 P c902805060df4df9b5599d0e94b2dc06 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:37:33.453900 19032 consensus_peers.cc:485] T 8937c809610542d88fdde3daf002c890 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:37:33.480985 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:33.485906 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:37:33.505244 18900 consensus_peers.cc:462] T 246a8425400f4890a5264c4ae8a627c5 P c902805060df4df9b5599d0e94b2dc06 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111)
I20240502 14:37:33.507076 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:33.509810 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:37:33.510066 18637 consensus_peers.cc:485] T d28db16c9f7e405b8866f0e6c548e246 P 2db304b538e14857b7b1cc272d0db94b -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:37:33.524266 18900 consensus_peers.cc:485] T ecb8a9b04ba34f779e2227e77e6f764c P c902805060df4df9b5599d0e94b2dc06 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:37:33.560088 19032 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:37:33.593071 18769 consensus_peers.cc:485] T 9580f3fdf7084cc0923b1b16bfedd450 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:37:33.599539 19031 consensus_peers.cc:485] T 097eb705c9e24740a5be61bf3354f3d8 P 007b2e79dc424a6f80141180f072f2d8 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 14:37:33.732646 19077 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:33.747270 18681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:37:33.766357 18812 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:37:33.771000 29954 raft_consensus.cc:998] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b: Attempting to remove follower ff6acba046404ef0a6f41ad5e09d852f from the Raft config. Reason: Leader has been unable to successfully communicate with peer ff6acba046404ef0a6f41ad5e09d852f for more than 300 seconds (300.375s)
I20240502 14:37:33.771833 29954 consensus_queue.cc:237] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } }
W20240502 14:37:33.801185 18768 consensus_peers.cc:485] T 2c2afaffde96462eacd0d32bbdec23b1 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn't send request to peer ff6acba046404ef0a6f41ad5e09d852f. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.72:35871: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:37:33.802767 18965 raft_consensus.cc:1261] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:37:33.803611 29953 consensus_queue.cc:1035] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 14:37:33.817512 18766 consensus_peers.cc:462] T 414e03d597934da3a6a74d786aeab134 P fec04c50dc5b49878ffa3c3873df32e4 -> Peer a14c220ad2cc4bf9bd7dad65f4803eb0 (127.5.107.70:46021): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.107.70:46021: connect: Connection refused (error 111)
I20240502 14:37:33.820925 29748 raft_consensus.cc:2932] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
I20240502 14:37:33.830924 18556 catalog_manager.cc:5484] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b reported cstate change: config changed from index 9 to 10, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) evicted. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
I20240502 14:37:33.833922 18965 raft_consensus.cc:2932] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } }
I20240502 14:37:33.837994 18701 consensus_queue.cc:237] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } }
W20240502 14:37:33.840847 18634 consensus_peers.cc:485] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b -> Peer fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66:33009): Couldn't send request to peer fec04c50dc5b49878ffa3c3873df32e4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:37:33.847472 18962 raft_consensus.cc:1261] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Refusing update from remote peer 2db304b538e14857b7b1cc272d0db94b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:37:33.848623 29954 consensus_queue.cc:1035] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:37:33.860070 30040 ts_tablet_manager.cc:902] T b15d11209e45473e863766e5cd6e822f P fec04c50dc5b49878ffa3c3873df32e4: Initiating tablet copy from peer 2db304b538e14857b7b1cc272d0db94b (127.5.107.65:36749)
I20240502 14:37:33.867354 29748 raft_consensus.cc:2932] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } }
W20240502 14:37:33.872411 19032 consensus_peers.cc:485] T 6a992d12aee7416096d1353533397386 P 007b2e79dc424a6f80141180f072f2d8 -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:37:33.873859 18544 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 0ffb6e582ef04fb38dd273c13febac58 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:37:33.876367 18962 raft_consensus.cc:2932] T 0ffb6e582ef04fb38dd273c13febac58 P c902805060df4df9b5599d0e94b2dc06 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } }
I20240502 14:37:33.877040 18556 catalog_manager.cc:5484] T 0ffb6e582ef04fb38dd273c13febac58 P 2db304b538e14857b7b1cc272d0db94b reported cstate change: config changed from index 10 to 11, VOTER fec04c50dc5b49878ffa3c3873df32e4 (127.5.107.66) added. New cstate: current_term: 1 leader_uuid: "2db304b538e14857b7b1cc272d0db94b" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2db304b538e14857b7b1cc272d0db94b" member_type: VOTER last_known_addr { host: "127.5.107.65" port: 36749 } } peers { permanent_uuid: "c902805060df4df9b5599d0e94b2dc06" member_type: VOTER last_known_addr { host: "127.5.107.67" port: 45409 } } peers { permanent_uuid: "fec04c50dc5b49878ffa3c3873df32e4" member_type: VOTER last_known_addr { host: "127.5.107.66" port: 33009 } } }
I20240502 14:37:33.886286 18945 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:37:33.887399 30040 tablet_copy_client.cc:308] T b15d11209e45473e863766e5cd6e822f P fec04c50dc5b49878ffa3c3873df32e4: tablet copy: Beginning tablet copy session from remote peer at address 127.5.107.65:36749
I20240502 14:37:33.888062 18721 tablet_copy_service.cc:140] P 2db304b538e14857b7b1cc272d0db94b: Received BeginTabletCopySession request for tablet b15d11209e45473e863766e5cd6e822f from peer fec04c50dc5b49878ffa3c3873df32e4 ({username='slave'} at 127.5.107.66:41693)
I20240502 14:37:33.888192 18721 tablet_copy_service.cc:161] P 2db304b538e14857b7b1cc272d0db94b: Beginning new tablet copy session on tablet b15d11209e45473e863766e5cd6e822f from peer fec04c50dc5b49878ffa3c3873df32e4 at {username='slave'} at 127.5.107.66:41693: session id = fec04c50dc5b49878ffa3c3873df32e4-b15d11209e45473e863766e5cd6e822f
I20240502 14:37:33.890776 18721 tablet_copy_source_session.cc:205] T b15d11209e45473e863766e5cd6e822f P 2db304b538e14857b7b1cc272d0db94b: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:37:33.891988 30040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b15d11209e45473e863766e5cd6e822f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:37:33.900394 30040 tablet_copy_client.cc:783] T b15d11209e45473e863766e5cd6e822f P fec04c50dc5b49878ffa3c3873df32e4: tablet copy: Starting download of 5 data blocks...
I20240502 14:37:33.906078 30040 tablet_copy_client.cc:647] T b15d11209e45473e863766e5cd6e822f P fec04c50dc5b49878ffa3c3873df32e4: tablet copy: Starting download of 1 WAL segments...
I20240502 14:37:33.908093 30040 tablet_copy_client.cc:523] T b15d11209e45473e863766e5cd6e822f P fec04c50dc5b49878ffa3c3873df32e4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 14:37:33.909080 18637 consensus_peers.cc:485] T df5f14206a1d46e4866f34fa234f945b P 2db304b538e14857b7b1cc272d0db94b -> Peer a834edffbcb2469aae598b069aab0681 (127.5.107.69:41119): Couldn't send request to peer a834edffbcb2469aae598b069aab0681. Status: Network error: Client connection negotiation failed: client connection to 127.5.107.69:41119: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:37:33.911595 30040 tablet_bootstrap.cc:492] T b15d11209e45473e863766e5cd6e822f P fec04c50dc5b49878ffa3c3873df32e4: Bootstrap starting.
W20240502 14:37:33.928038 18636 consensus_peers.cc:485] T b315327141954b53a0f87c2d1dc14b9b P 2db304b538e14857b7b1cc272d0db94b -> Peer ff6acba046404ef0a6f41ad5e09d852f (127.5.107.72:35871): Couldn'